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SYNOPSIS 


Bie validity of certain statistical procedures depends 
oa the properties of the method of maxirttn-likelihood, viz., 
its insistency and the fact that it leads to an estimator 
^ose asymptotic variance can be derived easily from the 
likelihood function. These properties are well established 
for the case where successive observations are independent 
Slid identically distributed. This will not always be the 
case in practice. For example, the random variables may be 
related by a stochastic difference equation in ^diich case 
the observations are. not independent. Since the error terms 
involved in the difference equation need not always have the 
same distribution, the observations may not even be identically 
distributed. These instances are the itKJtivations for the 
study of asymptotic properties of the maximum likelihood 
estimators (MLE) in some general cases. Considerable work 
has been done in this direction to cover the case of 
stationary Markov processes and the case when the 
ohsearvations are independent but not idantically distributed. 

In this investigation ve consider the problem of obtaining 
khe asymptotic properties of Ml£ for the case of dependent 
random variables. This is done in three different cases, 
namely , when the observations are frcsn Ci> stationary 



tt>^peii(3en1: sequences of random variables ^ (ii) stationary 
^-mixing parocesses , and (iii) arbitrary stochastic processes. 

The present thesis is divided into five chapters. A 
brief review of work done upto now in the area of maximum 
likelihood estimation irelevant to our discussion , and a 
summary of the results obtained in this thesis are given 
in Chapter I. 

We obtain weak consistency and first-order efficiency 
of MI»E for stationary m-dependent random variables in 
Chapter II . Weak consistency is proved using the Bernstein ’ s 
method of grouping randcmt variables ^ich he used to obtain 
the central limit theorem for dependent remdom variables. 
Under the set of regularity conditions presented in this 
Chapter, we are \mable to obtain asymptotic normality, 
Bcwever, satoe ccmm^nts are made about the asymptotic 
distribution. An example , satisfying the regularity 
conditicais ^ is given. 

In Chapter III, this problem is considered for 
stationary 4»~®ixing processes. We prove a form of weak 
law of large numbers and central limit theorem for double 
sequences of randcmi variables. Under the set of conditicms 
presented in this chapter , these results are used to prove 
weak consistency and asymptotic normality of an approximate 
maximum likelihood estimator (AM/XE) ’idiich is similar to the 
usual MLE. Exas^les satisfying the conditions are given. 
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A more general prc^leiB of establishing strong 
consistency, asymptotic normality and first-order 
efficiency of MLE in the case of arbitrary stochastic 
process, is discussed in C3iapter IV, !l3ie regularity 
conditions are expressed in terms of probability 
distribution of observations conditioned upon all past 
observations. As an application of these results, the 
Bernstein-vonMises theorem regarding the convergence of 
posterior density to normal is proved. We use this 
theorem to establish strong consistency , asymptotic 
normality and some other asymptotic properties of Bayes 
estimators. Examples are provided to illustrate the 
theory of this chapter. 

Chapter V includes some ccanments on our work and 
suggestions for further research in this important field. 



CHAPTER I 


INTRODUCTION AND SUMIIARY 


The method of maximum likelihood estimation is in 
common use for many years. The nethod can be described as 
follows. For a specified statistical model, given the sample 
values of , that is, (x^, . = . ,x^) , the likelihood 

function is a function on the parameter space 0 or on the 
family ./“'of probability measures which, for a given value 
of 6 (respectively P) , is proportional to the (probability) 
density function of , . . . , Xj^) at (x^^ , . • . , Xj^) . The 
factor of proportionality may depend on (x^^, . . . ,Xj^) . Thus ^ 
if p(x 2 , ..., Xj^ ; 0) is the joint (probability) density 
function of (X^ , •*. , X^j) under 6, the likelihood function 

of 6 , given (x^^ , . . . , Xj^) = Xj^ , is 

X(6i 3^) = K(Xjj) p(x 2 ^, ..., x^? 9), 9 e 0 

A maximvm likelihood estimator {'IIE) of 9 is defined as a 
point in ©, if such a point exists, for which X(6j x^^) 
attains its suprsmum. Roughly speaking, one could say that 
the probability of observing a value in a close neighbourhood 
of the sample value is maximised when the state of nature 
is a maximum likelihood estimator. It is to be noted here 
that X(8j X ) does not have to be differentiable with 

<vl* ' ' 


respect to 0 . 
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Maximum likelihood method serves as a basis for large 

sample procedures of testing hypotheses and of confidence 

intejTval estimation. Moreover, maximum likelihood estimators 

possess the invariance property , namely , if 9^ is an M.LE of 6 

then g(8^) is an MLE of g(0) for any function g{6) of 6 {not 

necessarily one-to -one) . Furthermore , under some regularity 

conditions, maximum likelihood estimators 6^ of 9 have some 

desircible properties such as consistency (that is , 8^ 

converges to the true parameter value 9^ of 9 either in 

probability or almost surely as the sample size n tends to 

infinity) , asymptotic normality (that is the asymptotic 

distribution of 9^ is normal) , and asymptotic efficiency 

(in the sense that the covariance matrix of the asymptotic 

distribution of 6 is the inverse of the Fisher information 

n 

matrix) . 

Ihe concept of asymptotic efficiency in the sense 
given above, which was called by Wald [50] as strict sense 
efficiency , is based on the covariance matrix of the 
asymptotic distribution of the estimators. Since the 
estimators may not have limit distributions, Wald [50] 
introduced the concept of v/ide sense efficiency which 
reduces to the. usual strict sense efficiency whenever the 
limiting distribution exists and is normal. 

Examples of estimators with asymptotic variances 
which never exceed the inverse of the Fisher information 



3 


and at some parameter points are below it were provided by 
Hodges and reported by LeCam [24] . Such astiraators are 
called super" efficient (for example, see Zacks [53] p„208) . 
LeCam [24] has shown that the sot of parameter points, on 
which a super-efficient estimator has an as}/mptotic variance 
smaller than the inverse of the Fisher information , is of 
Lebesgue measure zero. Rao [37] investigated the problems 
of asymptotic efficiency in a series of papers. Ha showed 
that super-efficient estimators , among consistent 
asymptotically normal ones, can exist if the approach to 
asymptotic normal distribution is not uniform on compact 
sets of the parameter points. Such a result was also 
established by Bahadur [3] . Thus , Rao suggested that one 
should consider only those consistent asymptotically nomal 
estimators whose convergence in distribution to noirmal is 
uniform. Ha has shown that, under certain regularity 
conditions , the asymptotic variance of such estimators is 
always at least as large as the inverse of Fisher information. 
Ihus, the uniformly best asymptotically normal estimators are 
the most efficient in this restricted class of estimators. 

Such estimators are called first-order efficient estimators 
by Rao. 

The asymptotic properties of maximum likelihood 
estimator have been studied by many people under a variety 
of conditions. Cramer [14] gave proofs of weak consistency. 
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asymptotic nojrmality and asymptotic efficiency of MLE in the 
indarxsndent case, iindar some regularity conditions. !Ehese 
regularity conditions include the existence of the first 
three derivatives of the log-likelihood function and some 
uniform integrability conditions. An outline of the proof 
of strong consistency of the HLE was given by Wald {51] and 
^‘folfowitz [52] , but on the assumption that the estimate 
really maximises the likelihood function. Hxiber [19] 
established these results for the families of densities 
which do not satisfy the usual differentiability assumptions. 
Prakasa Rao [34] showed that MLE is strongly consistent and 
hyper-efficient for a special class of densities which do not 
satisfy the usual regularity conditions of Cramer [14] . 
Conditions under which MLE is uniformly consistent were 
presented recently by Moran [29, 30]. Ibragimov and 
Has'minskii [23] proved the asymptotic no 3 ntiality of the MLE 
without the assimiptions of continuity of log-likelihood 
function and existence of the second derivative of the 
likelihood function. In all these cases, it is assumed 
that the observations, on which the MLEs are based, are 
independent and identically distributed. 

These results have been extended in many directions. 
For example, some results were obtained for models in which 
the observations are independent but not identically 
distributed. It is worth mentioning that, under this 
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situation, even if the observations are (non identically) 
normally distributed, the MLE may not be consistent and yet 
a consistent estimator does exist (see Zacks [53] , p.236) . 
Chao [12] established the strong consistency of MLE when the 
observations are independent but not identically distributed. 
In the same situation ^ Hoadley [18] gave conditions under 
which MLE is weakly consistent and asymptotically normal. 

His approach to consistency is similar to that taken by 
Wald [51] . 

Billingsley [8] proved weak consistency and asymptotic 
normality of an MLE for observations from a Markov process 
with stationary transition measures. His assumptions are 
analogous to those imposed by Cramer [14] in the case of 
independent randcan variables. The proof of asymptotic 
normality is based on a central limit theorem for martingales 
due to Billingsley [7], For the same case, Roussas [42, 43] 
obtained strong consistency and asymptotic normality of the 
MLE, xinder different conditions. Some results are also 
obtained by Prcikasa Rao [35] generalizing the results of 
Huber [19] . Sarma [45] proved that an MLE is also first- 
order efficient in addition to tlie properties of consistency 
and asymptotic normality. 

Ihe asymptotic properties of maximum likelihood 
estimators for the case of arbitrary stochastic process 
were first studied bjf Wald [50] . He gave conditions under 



which HLE is weakly consistent and proved that a consistent 
estimator is asymptotically efficient at least in the wide 
sense. 11. M. Rao [39] extended these results to non-stable 
processes. Using a version of the central limit theorem 
due to P.Le''7y, Silvey [48] pointed out the conditions under 
which an MLE is asymptotically normal. He gave conditions 
for weak consistency , but they were not expressed in terms of 
the probability distribution of the observations, and 
therefore, do not appear to be readily applicable. In a 
more recent work, Bar-Shalom [4] presented a set of 
conditions which ensure weak consistency and asymptotic 
efficiency of MLE in the case of dependent random variables. 

Under a set of conditions which is stronger than that 
for consistency of MLE, LeCam [24, 26] proved the consistency 
of Bayes estimator for eveary parameter point in the 
independent case, Lindley [27] has established heuristically 
that MLE and Bayes estimators are asymptotically equivalent, 
LeCam [25] proved one of the fundamental results in the 
asymptotic theory of inference viz. , the Bemstein-vonMises 
theorem regarding the convergence of posterior density to 
normal. Special cases of this theorem were first obtained 
by S. Bernstein and R. vonMises {for reference see LeCam[25]}. 
Bickel and Yahav [6] and Cliao [13] discussed the asymptotic 
behaviour of Bayes estimators . Itecently , Ibragimov and 
Has 'minskii [21, 22] studied the limiting properties of 
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Bayes estimators and maximum likelihood estimators under a 
different set of assumptions, Borwanker, Kallianpur and 
Prakasa Rao [11] extended the results of Bickel and YahavfS] 
and LeCam [24, 26] to the Markov case. 

In this thesis the asymptotic properties of maximim 
likelihood estimators and results concerning Bayes estimators 
are obtained in the case of dependent random variables. This 
is done in three different cases , neimely , when the observations 
are fr<mi (i) stationary m-dependent sequences of random 
variables, (ii) stationary <{> -mixing processes, and 
(iii) arbitrary Stochastic processes. 

Conditions are given which ensure weak consistency 
and first-order efficiency of MLE for stationary ra-dependent 
random variables, in Chapter II. The approach adopted is the 
method of Bernstein, which he used to prove central limit 
theorem for m-dependent random variables. We were unable to 
give conditions for asymptotic normality in this case. But 
some comments are made about the asymptotic distribution. An 
example satisfying the regularity conditions in which !1LE is 
asymptotically normal is given. 

In Chapter III, weak consistency, asymptotic 
normality and first-order efficiency of approximate maximum 
likelihood estimator (AMLE) (for definition, see Section 3.5) 
are obtained for stationary 4>“®aixing processes . An 
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alternative approach based on conditioning on past observations 
is used in this chapter. Under the set of conditions presented 
in this chapter 5 the proofs of weak consistency and asymptotic 
normality of AMLE depend on vreak law of large numbers and 
central limit theorem for double sequences of random variables. 
Weak law of large numbers for double sequences of randcan 
variables is proved using Khintchine's method. 'Hiis method 
was also used by Rosen [40] to prove central limit theorem. A 
particular case of the central limit theorem of Rosen [40] for 
double sequences of stationai^r 4niiixing random variables is 
used to prove the asymptotic normality of AMIiE. Examples 
satisfying the conditions are given. 

Strong consistency, asymptotic normality and first- 
order efficiency of MLE are proved in Chapter IV for 
arbitrary stochastic processes. The regularity conditions 
are expressed in terms of probability distribution of 
observations conditioned upon all past observations, as in 
Bar “Shalom [4] , In this case, we have also obtained 
Bernstein “VonMises theorem. An example is provided. 

Crcimer's approach is used in all the above results. 
Alternative possible approaches are the method of contiguity 
as given in detail in Roussas [44] or the method of 
convergence in distribution of stochastic processes as used 
in Prakasa Rao [34] and Ibragimov-Has 'minskii [23]. 
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CSiapter V includes some comments on our work and 
suggestions for further research in this important field. 

Whenever we need to refer to an equation C or a 

theorem C or a lemma C ,, in Section B of Chapter A, we do 

so by writing (A.B.C.; or theorem A.B.C. or lemma A.B.C., 

respectively. Throughout the following chapters- we write 

(p)lim, (a,s)lim and (<i-) lim to denote that a seqtience 

n-^ 

of rauidom variables converges in probability, converges 
almost surely and converges in distribution respecti'roly. 



CHAPTER II 


I4AXIMUM LIKELIHOOD ESTI^mTION FOR 
STATIONARY m-DEPEHDENT SEQtBNCES 

Introduction 

Successive contributions by many authors ^ including 
Cramer [14], Wald [51], Hx±>er [19], Prakasa Rao [34] and 
Moran [29,30] have yielded sufficiently weak regularity 
conditions under which the consistency ^ asymptotic noinnality 
and asymptotic efficiency of a maximum likelihood estimator 
(MLE) hold for independent and identically distributed random 
variables. These properties of MLEs are also studied for 
independent but not identically distributed random variables 
(see Chao [12] and Hoadley [18]) and for stationajry Markov 
processes (see Billingsley [8], Sarma [45], Roussas [42,43] 
and Prakasa Rao [35]) . The main tools used to prove 
consistency and asymptotic normality in these cases are 
laws of large nxambers and central limit theorems respectively, 
for sums of random variables . So it is obvious that these 
results will extend to more general stochastic processes for 
which laws of large numbers and central limit theorems hold. 
Restalts of this type are available under various conditicais 
on the type of dependence between the random variables (see, 
for example Ibragimov [20], Rosen [40] and Serf ling [46]). 
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t < t2 < . - . < tg 

subset { 

(tj, - tg) > m. 


is stochastically independent of auiy other 
. . . } , tg < tj^ < tj^+2. * * * ' whenever 


Let {Xj^,n ^1} be a stationary sequence of random 
variables and ^ be the a -field generated by this sequence. 

Let T be the shift transformation corresponding to the 
stationary process {Xj^z n >1} {for details of these concepts, 
we refer to Doob [16] , p. 452) . Then we have the following 
lemmas : 

Lemma 2 .2 . 1 A necessary and sufficient condition for a 
stationary process {X^ / n > 1} to be ergodic is that for any 
two sets A and B nffiasurable with res,pect to 
, n-1 

lim n”^ \ P(B Q T~3 a) = P(A) P(B) . (2.2.1) 

n-i^o j=o 

Proof i For proof of this lemma , we refer to Rosenblatt [41] ^ 

p. 110. 


Remark 2.2.1 t Equation (2.2.1) is implied by a stronger 
condition , namely 

lim P{B n T“3 A) = P{A) P(B) (2.2.2) 

j-K» 

for any two sets measurable with respect to This conditicai 
is commonly referred to as mixing condition and is. essentially, 
a form of asymptotic independence. 
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Lemma 2.2.2; Let ^ be a field generating the 0 -field^, 
if (2.2.2) holds for any pair of sets A and B in^^, then 
it holds for any pair in 

Proof: For proofs see Iheorem 1.2 of Billingsley [9], p.l2. 
Lemma 2.2.3 : Let {35^, n k 1) be a sequence of stationary 
m-dependent random variables . Then it is ergodic. 

Proof: Let ^ be the a -field generated by the seqtjence 
{Xjj, n ‘k. 1} and T be the shift transformation corresponding 
to this process . Let be the field generated by the random 
variables X 2 ^, ..., Then^ is the 0 -field generated by 

00 

the f ield ^ ^n* lemma is proved, if we show that 

n=l 

(2.2.2) is satisfied for any pair of sets A and B measvirable 
with respect toiS. However, in view of Lenana 2.2.2, it is 
enough to verify (2.2.2) for any pair of sets A and B in 
n > 1. Let A and B be the sets 

A = {m ; X2^(a)) < a^^, — , - ^n^ ^ (2.2.3) 

B {m : X2^(w) ^ » o . , Xj^(w) < bj^} . (2.2.4) 

where a^ and b^, 1 < i < n are real numbers. Then 

T~3 a ={u ; X3^^j(ii)) ^ ..., Xj^+j(m) < a^^} . (2.2.5) 

"1 

For j >n+m-l, the sets B and T~ A are independent 
because of m-dependence of the sequence n i 1} of 

random variables. Further P(T”^ A) = P(A) , since the 
sequence is stationary. Iherefore, for j > n + m - 1, 
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P(B D A) = P{B) P{A} (2.2.6) 

so that (2.2.2) is satisfi -d for A and 3. Hence the leimna 
follows . 

Definition 2.2.2 ; A sequence of random variables 

n 2: 1} is said to be bounded in probability if, for 

any e> 0 , there exists an M such that 

£ 

lira sup PElXjj} >M ] < e. (2.2.7) 

n oo ^ 

The following lemma is well-known. 

Lemma 2.2.4 ; Let {X^ , n > 1} and {Yj^, n t 1) be two arbitrary 
sequences of random variables. Then the following results 
are true . 

(i) If (p) lira = X and f is a continuoxis function on R, 

n-^ 

then (p)lira f(X-) = f(X). 
n-^ 

(ii) If (p) lira X = 0 and {Yj^} is bounded in probability, 

n-xM 

then + Yj^} is bounded in probability. 

(iii) If (p) lira X = 0 and {Y } is bounded in probability, 

then {p)lim X^Y^ =0. 
n-x» 

(iv) If (p)lira X_ = c(^ 0) and {Y„} is bounded in 
probability, then (Yj^/X^} is bounded in probability. 

(v) If (p)lim (X-j - Y_) = 0 and X_ converges to a random 
variable X in distribution or in probability, then Y^^ also 
converges in distribution to X, 
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Proof: Tile proof of (i) can be found in Tucker f491 p.l04. 
For the proofs of remaining results see M.M.Rao [381 and the 
references there. 


Lemma 2,2.5 : A sequence of jointly distributed random 
variables with zero mean and uniformly bounded variances 

obeys the strong law of large numbers, in the sense that 
P[liia Zjj = 0] =1, if positive constants M and q exist such 

n-Ko s 

that for all integers n. 


|E{Xn Z^)\ i M n”*^ , 

I 

^ n 

where = n"-*- ^ Xj^. 

i=l 


C2.2.8) 


Proof : For proof, we refer to Theorem 2B, Parzen [32J p.. 420. 
Lemma 2.2.6 : Let n = 1, 2, ..., k? k i 1} be a double 

sequence of random variables satisfying the following 
conditions : 

(i) The random variables X" are m-dependent for any fixed 


k k 1 , 

(ii) E{2^) = 0 for all n and k^ 

(iii) Var(S]^) =1, k > 1, v/here 

k 

(iv) limsup ^ Var(3^) < «>, 

Jc-Ko n=l 

(v) lira T / (x) = 0 

k-~. nil- |x)>e 


I ^ 


n=l 


n 


for every e > 0 where is the distribution function 

of 3^. Then converges in distribution to normal with 


mean zero and variance one. 
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Proof; For proof see Theorem 13.1 ^ Rosen [40 J . 

2*3 Segularity Conditions ; 

Let n 1} be a stationary m-dependent sequence 

of random variables. Let plx^ denote the n- 

dimensional joint density function which depends on a single 
unknown parameter 8. The problem is to study the asymptotic 

irN 

properties of an MLE 9^ of 6. 

We denote pCx^^,.. . . ,x^ ; 0), (|)j^{xj^, . . . ,Xj^;6) and 

P‘-'*(i-l)k+r+l ‘'y Pn<®*' ♦k'®’ H,k(9! 

respectively in this chapter. We write ?{<■} and E{»} 

instead of {-land Eg {»} respectively, 
o °o 

EJefinition 2.3.1; An estimator e„ e 9„ (xn , ..., x_) of 
the unknown parameter Q, which maximises log pCx^^,.. . ,x^;6j 
in a neighbourhood of the time parameter value 6^, is said 
to be a maximum likelihood estimator (>4LE) of 6. 


Suppose the following regularity conditions are 
satisfied by the n-dimensional joint density function • 

(HI) The parameter space 0 is an open interval in the real 
line and the true parameter value 6^ is an interior point 
of 0. 


1 r^, 

^^^2) log pj^{e)a»d L— log Pn(0) exist and are continuous 

86 

almost surely for all 9e8 and for all n. 
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(H3) Suppose that for all integers k > 2m, whenever n is 
of the form vk+r. 


V 

i_l 


(2.3.1) 


(this representation is useful in view of m-dependence of 
the sequence of random variables n > 1 }) where Pj.(9) 
and 4>jj.(9) satisfy the following conditions for all 8 e 0 . 


(i) 

(p) lim 

(p)lim 

^ ^ 0 

log 

P_(6) = Qr 

( 2 , 

.3.2) 


k-^ 


36^ 


J. 



(ii) 

(p) lim 

(p) lim n“^ 

3 

log 

4>^(e) = 0, 

( 2 . 

.3.3) 




36 


Jv 



and 









(iii) for each 8 e 0, there exists a neighbourhood Vj^{9) 
of 9 such that 

{p)lim (p)lim C sup { n“^ -if- log = 0 . ^ 

k-x» n->o° 6'eVj(6) 36^ 

...(2.3.4) 

(H4) For all Gee, 

Eg {L- log Pn(e)> = 0 ^ for all n, (2.3.5) 

8 9 

E {S_ log p„(e)}^ = - E-{-^ log p„( 8 )} 

® 36 ® 39 ^ ^ 

= ljj(9) < “f for all n, (2.3.6) 

and 

lim n“^ 1 ^( 8 ) = 1(6) (2.3.7) 

n-Kso 


where i( 0 ) is finite amd positive. 
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(H5) For every 6 e 0, there exists a neighbourhood V{9) 
of 6 such that, for every 0' e V{9) 

j^logp^(9) -^logp^(0')l i l9-e’l 

where G{x^, . . . ,Xj^) is non -negative with 
[G(x^, . . . 3 = M^(9), finite. Further, 

lim n“^ Mj^{8) = M{0) < «>. (2.3.8) 

n^^ 

Hereafter, we denote log p (0) by i|j (0) , log 4*^,0) 
by hj^(0) and log j^(0) by g^ ^(6) . If f{0) is a function 
differentiable twice with respect to 8 , we denote the first 
and second derivatives of f<0) by f’{9)and f"(0) 
respectively. Unless otherwise specified all statements 
v;hich involve convergence of random variables in the following 
section are valid with respect to the true probability 
nKaasure . 

2 The Asymptotic Properties of a MLE; 

We prove in this section the consistency and first- 
order efficiency of a MLE. 

Theorem 2.4.1 : Under the assumptions of Section 2.3, the 

Weot-k.ty 

likelihood equation has a root which is^consistent. 

Proof; Expanding fi(9) = ^ log p„(6) about the true value 
— — — “ 38 “ 

8q of the paranteter 6, we obtain that 
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^^(8) = ^^(0^) + (e-e^) 


+ {8~e^) {fj,{6’) - 


(2.4.1) 


where 9' = 6- + a(6-6 ) and [aj < 1. By (2.3.1) the right 


■'o ■ '^o' 

side can be written in the form 


{*^( 80 ) + hjj(e^) + I 


V 


+ (8-e^) {♦;(6o) + hj;(e^) + I gj re )} 

i=l '*• 


+ (6-60) [{>j^;:-(e’) - if^::(9^)} + {h-on -h^(8Q)} 


V 




It follows firom (H5) that there exists a neighbourhood 
V(0 q) of 8^ such that for 6' e V(8^) 

|ti^J(6>) -4'"(8o)! < la(e-e^)| G(x^,...,)cp 


and 


Ig? 

i 1 


(e«) 




S la (9- 


■vi 


G(x 


{i-l)k+r+l'** 


'^ik+r^ 


where G(x^,...,x^) and G(x are 

non-negative, E[G(X^,..., X^) ] = M^(9^) < 

E[G(X^, Xj^)) = Mj^{8^) < and 

lira k"^ E[G(X 3 ^, ..., Xj^)] = M(0^) < «. 
k-x* 

Hence we get that, 

„-l ( 8 ) = + (6-9^) + (e-eo) 2 a^B'^> 


(2.4.2) 
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where | a j < 1 for all n , and 
n 


(n) 


-1 


®o“k = + ^.*(e )+ I q\ , C0^)} 

0,K r O <0 o 


+ (d-6) n“^ {h.Me’) -h"{0^)}. 

o .1 k ^ 


(2.4.3) 


B 


(n) 

l,k 


= (r (6^) + h;(e^) + J 


n“3.-^» (8 ) 
o' 


k'^o' 


i=l 


i,k' o' 


and 


®2^k “ {GCxj^, ... , x^) 


(2.4.4) 


^ iIl^^^Ci-l)k+r+l' •'*' ^ik+r^^* (2.4.5) 


Since {X^» n > 1} is a stationary m-dependent seqiience 
of random variables ^ for any fixed k >2nj and any fixed r 
such that o < r < k, ^g- ^j^.{9o> » i > l>x ^ ^ 

and ""ik+r^ ' i > 1} are stationary 

1 -dependent sequences of random variables and hence, by 
Lemma 2.2.3, are ergodic. Purtheraore, by (H4) and (H5) , it 
is clear that 

E[g* (0)1=0, 
l,k o 

and 

EfG(Xj^, ..., 



we get 


Therefore, by ergodic theorem, 

(a.s)lim V ^ ^ gr , (0 ) = E[a' (e )] = q 

{a.s)lim v“^ I g“ (9 ) = gj „ , (e )] = _ t fA \ 


and 


V 


{a,s}lim v"^ I G{X,. ,,, y . 

v-»<“ i=l (i-l)k+r+l' ''-f "iic+x"' 


E fG , ^r+k^ ^ 


= E[G(X. 


1 ' • ♦ • , Xj^ ) 3 




Since V -> €50 » 


s n -*■ «>, we. have 

V 


Ca.s)lin. n“ | '^(e ) = k”^ Efg*. (e )] 

n-xx. i-i i,k o ^^l,k^^o^-’ 


= 0 , 


, V 

{a.s)liin n--^ I g- (q y = j,-l j , j 
n-x» i=l o k' o' 


(2.4.6) 

(2.4.7) 


and 


(a. s) lira n”-*- T G(X,. y . 

n-^ it I (i-l)k+r+l' ^ik+P 

= k~l I 4 j^( 0 o} . ( 2 . 4 . 8 ) 

Therefore, it follows from assumptions (H4) and {H5> that 

V 

Ixm (a.s)lim n”-*- T cr* {a > = n 
k-+« jt-.^i.R®o^ 


4_1 i,R O' 

1=1 




(2.4.9) 


(a.s)liin n”-^ J g- (g ) = - i 

k-»<» istl i,k G o' 


(2.4.10) 
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and 

1 ^ 

lim <a.s) liinn" I H.iyy^+r+lK’ ' ' ' ^ik+r^ " 
k-x» n->® i=l 

...{2.4.11} 

where ~ f e > 0, 

P[n~lliJ)^(e^) 1 > el <£{’4-^(6^)]^ (ne)"^ 

= (ne)~^- 

Since I^Oq} is finite, the right side converges to zero as 
n •> “. Hence we obtain that 

lim (p)lim n“^ ^r^®o^ =0. (2.4.12) 

|^->oo Ti’-^ 

By (2.3.3) 

{p)lim (p) lira n~^ h/{0 ) =0, (2.4.13) 

n-*<o ^ 

and by (2.3.4), there exists a neighbourhood V^(8 q) of 8^ 
such that 

(p)lim (p)liin [ sup n“^ h^(6’)] = 0. 

]c+«> n-x* 0'eV2^(8^) 


Therefore 

(p)lira (p)liTn n“^ {h"(0') -hf(8_)} = 0. (2.4.14) 

From (2.4.9), (2.4.12) - (2,4.14), it follows that 

(2.4.15) 


0, 


(p)lim {p) lim * 0. 

By (2.3.2), we have 

(p)lim (p)lim n“^ = 

"k-*^ n-^^o ^ 


(2.4.16) 
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and by (2 .3. 4) , 

(p)lim {p)liii n~^ h"{6 ) =0. {2.4.17} 

Hence from (2.4.10), {2.4.16} and (2.4.17) we get 

(p)lim {p)lim = (p) lim n~^ p"{0 ) 

= - i^. {2.4.18} 

By Tchebyshev’s inequality, for every e > 0, 

P[n"^ G{Xj^, ..., Xj.)> e] < (ns) E [G{X^, . . . ] 

= (ns) Mr^®o^ * 

Since M^(9 q) is finite, the right side tends to zero as 
n Hence 

lim {p}lim n~^ G(X 2 ^^..., X^) =0. (2.4.19) 

k-»<» n-^ 

Therefore from (2.4.5), (2.4.11) and (2.4.19) we get that 

(2.4.20) 

5<0<6^+6}is contained 
— — o 

{2.4.21} 

{ 2 . 2 . 22 ) 


lim (p}lim B 2 ^ = M^. 

k-Ko n-xo ' 

Choose 6 > 0 such that {9 


in V{8 q) n 


(p)lim b; ^ = B ■ , 

n-^ 

(p)lim bI^'^ = k " 


(pjlim = ®2 ,k’ 


(2.4.23) 
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Here 

B - = 0 alnost surely for every k > 2in, 

(p)lim B, , = ^ i , 

]c-»-co ^ ° 

and 

M ( Qq) 

®2 k ~ "“v — — almost surely for every k >. ‘tyn. 

Note that and {B 2 are constant valued random 

variables with probability one. Let e > 0, Then there 
exists a positive integer k^ = such that, for all 

k > , , 


^fl®l,k + ^ol <5/2] < e/5, 


and 


!®2^i < (M^+l)/2 


with probability one. From (2 .4 .21) -{2 .4 .23) we get that 
for n > n^Ck^ ,e) , 

>6^1 S e/3, 

- B, , I > 5/2] < 6/6 , 


and 


(n) 




Let Si = I > 5^}, 

'^fko 


1 '■*“o,k 

^2 ^ ■^l®l,ko‘‘' ^ 
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®3 = ’ «o !>• 

Therefore, from (2„4.15>, (2.4.13) and (2.4.20) it follows 
that for every n > ,e) , P(Sj^) < e/3 for i = 1,2,3. Hence 

P(S) < e for n > n^Oc^ ^e) , where S = U For 6= 0^ + 6, 

(2. 4. 2) can be written as 


n-^ <,■ (e) = „„ 6‘ 


(2.4.24) 


For n > nQ(kQ,e>, 


''t|®o?k + “n B2X1‘ «^(«o+2)! -5 


> 1 - e. 


So if we choose 5 <iQ(MQ+3) , the sign of 

. _(n) . . _ fn) .2 


®o ~ ®2?ko ^ 


4. *» ' 

depends on that of -^Bi^j^with probability exceeding ..1-e 
for n > n^dcQ^). Therefore, for every n > n^dc^^e). 


i|,^{e} •> 0 for 0 = Sq - 5 

cind 

(6) <0 for 0 = + 6 

^n o 

with probability exceeding ..l-e. Since ^^{Q) is continuous, 
there exists a value 6^ in [0^ — 6, 0^ + 6] of 6, with 
probability greater than l-e for n > iiQ{kQ,e) such that 

’^n^n^ = le log Pn^®n^ * 
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15113 proves that 0 ^ exists aziK it is weak consistent since 

P(j 8 - 9_{ < S> > 1-e for n > n„{k„,e) . Hence the theorem 

HQ o o 

follows . 

Definition 2.4.1 • An estimator T„ of 0 is said to be 
as 3 ^ptotically efficient of first-order , if there exists 
two constants a and 6 o) , not depending on the sample but 
may depend on 0 , such that 

)| (2.4.25) 

converges to zero in probability as n -»-«>. 

Definition 2.4.2 ; A sequence of estimators of 8 is 

said to be asymptotically efficient in the wide sense , if 
there exists another sequence of random variables {W^^} 
such that 

(i) iim E 0 {Wj^) = 0 , 
n-*^ 

(ii) lim E {W 2 ) * 1 , 
n-^ 

and 

(iii) (p)lim {{lj^( 0)}^/2 ^ 0 ^ 

n-+« 

Remark 2,4.1 ; Suppose that (2.4.25) is satisfied for a 
sequence of estimators of 0 with a = 0 and 

8(0) = {Ijj(e)}^^^. I3ien (2.4.26) - (2.4.28) will be 

n 

satisfied with Wj^ log Pjj( 8 ). Hence if a sequence 

of estimators of 0 is asymptotically efficient of 


(2 .4. 26) 

(2.4.27) 

(2.4.28) 
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first-order, then it is asymptotically efficient in wide 
sense. 


Now, we prove the following theorem: 
theorem 2.4.2 ; Under assiomptions of Section 2.3, a consistent 
maximum likelihood estimator is asymptotically efficient of 
first-order. 

Proof; We have ^ 0 for large n since i_ > 0 by 

assumption (H4) . Hence, if 6^^ is a consistent root of the 
likelihood equation, then ^ from {2.4.1) , it follows that 

E^A<®o> + <Qn“®o^ ^^^^<^0^ ^ (2.4.29) 

where 

and j 0 * -6_ I < j 8„ -8 i . Since 6„ is a consistent estimator 
* o ♦ — * n o * n 

and $^(0) is continuous, by (i) of Lemma 2.2.4, we get 

(p) lim U = 0 . 

___ n 

Xi->oo 

Further^ (2.3.7) and (2.4.18) imply that 

<p)lim 

n-+<» 

Therefore, by (2.4.29), 

+ = 0 (2.4.30) 

where is a term which converges to zero in probability 
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as n ->■ “ and 

which tends to zero in probability as n -»■ ». {2.4.31) can be 
written in the form 


^n = ^n + Op{l) (2.4.32) 

where 

^n = {2.4.33) 

and 

1 /2 

f®n-3o>- (2.4.34) 

Since ECY^) = 0 and Var = 1, is bounded in probability. 

Then, {Y^-Op{l) } is bounded in probability by {ii) of 
Lemma 2.2.4 so that is bounded in probability by (iv) 

of Lemma 2.2.4. Now am application of (iii) of Lemma 2.2.4 
gives that 


(p) lim Z o_(l} = 0 


n-x» 


n p 


(2.4.35) 


Therefore, from (2,4.30) - (2.4.35), it follows that 

1 / 2 , 


{In(eo)}’^^^ iK^COo) 




= 12„ Op(l) + , (2.4.36) 

which tends to zero in probability as n ■*■«>, Hence the MLE 
of 0 is asymptotically efficient of first-order. 
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Remark 2.4.2% By (2.4.32) - (2.4.35), it follows that 0 is 
n 

also asymptotically efficient in wide sense in view of 
Remark 2.4.1. 

Remark 2.4.3 ; If the sequence converges in distribution 

to normal, then the efficiency in wide sense coincides with 
the classical strict sense concept, that is with Fisher 
efficiency (for definition see Cramer [14]). 

Under the set of assumptions given in Section 2.3, we 
are- unable • to prove the asymptotic normality of MLE, However, 
we can make the following remark about the asymptotic 
distribution of MLS. 

Remark 2,4.4 ; Since 0^^ is a MLE, we hav’-e (2.4.29) - (2.4,35). 
Therefore, 

(p)lim (Y^-Z^) = 0. (2.4.37) 

Hence if Y^^ converges to Y in distribution as n " then, by 
(v) of Lemma 2.2.4, it folloK^s that 2^ also converges in 
distribution to Y, Consequently, in order -to obtain the 
asymptotic distribution of Z.^, it suffices to find the limiting 

distribution of Y as n ->• “ if it exists. 

n ^ 

In general, if p(x-j^, . . , ,Xj^; 0} is of the form given 
in (2.3.1), it is difficult to give conditions iinder which 
Yjj converges to a normal distribution. M.M.Rao [39] proved 
that Y^ will have an asymptotic distribution if the 
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correlation between and tends to one nnifojrmly 

in r as n This condition is difficult to verify even in 

the m-dependent case. In some special cases , it can be 
shown that the asymptotic distribution of MLE is normal. One 
such example is given in the next section. 

2.5 Example : 

In this section, we give an example whi<di satisfies 
the conditions of Section 2.3. Let the sequence n > 1} 

of randcam variables be defined by 

^ = Vp <2-5.1) 

where n ^ 0} is an independent sequence of random 

variables, each being N(p,l) where y is unknovm and 
jp] < 1 is a known quantity. We assume that {X^^} is observed. 
This is the first-order moving average process. The problem 
is to obtain the asymptotic properties of an MLE of y. We 
show that the conditions of Section 2.3 are satisfied and 
thus establish the v/eak consistency and first-order efficiency 
of the MLE y^ of y. Fiirther ^ we prove that is asymptotically 
normal. It is clear from (2.5.1) that 


E(Xj^) = y(l-P) , 

(2.5.2) 

Var(Xjj) = 1 + p^. 

(2.5.3) 

Cov(Xjj,3^) = -P for jn^mj = 1 


= 0 for In-ml > 1 

(2.5.4) 


and the n-dimensional joint density function which depends on 
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the single unknown paraitteter y is given by P^ty) 


= { 2 Tr )^/2 


<dntr^) 


' 1/2 


1 ^ 
expC- j I 

i=l 


n . . 

I (x£-Cy) (x--Cy)J 
3=1 - . 2 


... (2.5.5) 


where C = .i-p and is the variance -covariance matrix of 

(Xj^, . . . . The elements of are given by (2.5.3) and 

(2.5.4) . is the i ,1 th element of r~^. Ihen we have 

‘n n 

(see Shaman [47}), 

Y^3 = pi"3{l-p23) (i-p2n-2i+2j {(i^p2j ^_p2n+2j j-l 

H 

...(2.5.6) 

for i > j. Prom (2.5.3) - (2.5.5), it follows that the 
seqtience {X^ , n > 1 } of random variables is stationary and 
1 “dependent . Now , we have 

n n . . 

~ log p (y) = C .1 I Yn^{Xi+x.-2cy) 

3y ^ 1—2 j-i “ J 


V X* 

=1C I Y„ (X. -CU) 
i=l ^ ^ 

and 

l^log p (y) = -c2x2.I Y^“ 

3y2 ^ i= 1 n 


where 


. n 


i-l 


n 

• • 

T^' = I 

n 5=1 

yij 

= I 

3=1 

^ij 

n 

* .1. 
3=3- 

-y.i-3 



i-l 


n 




= I 

3=1 


^ I. 

3=i 

n 


(2.5.7) 


(2.5.8) 
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We have 


i-1 . , i_l 

{p^~3(L.p2j) (i„p2n-2i+2, 

J-J- 3=1 

by {2.5.6) y where 


= dVXlV'^^) 


Therefore, we gat 


{2.5.9) 




- t(l-p)qjj}*^ p(l-p^'^) (1-p^) (l-p2“-2i+2) 


Similarly, by {2.5.6), we get 


ill “ f(l-p)q^)'^ (l-p2i) (i_pn-i+l, (^_^n-i+2j 


SO that 


fpX-p^“^) ( 1 -pi) (l_p2n-2i+2j 


+ (1-p^^) (l-p^“i‘*'i) (l-p””i'*’2j 


Xl-p)q^}-i (i-pi) (i„pnX+lj [p(i_pi-lj ji^pH-i+lj 


+ (1+p^) {l-p^“i+2. 


= {l-p^){l_(/'^-i+i) t 


(2.5.10) 


where 


= (l-p^+l) {(i.pj2^3^_p2n+2jj-l^ 


(2.5.11) 
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®ierefore, 

n 


I < 

i=l ^ 


a 




i=l 


= t^Ind+p""^^) - 2p(l-p^) (l»p>-^] 

= n{l-p)~^ - 2pt^ d-p"") 

Hence, by {2.5.8), we have 


n 21 

f-^logpj^Cy) = -2[n-2pt^ d-p’^) {l>p) 1 . {2.5.12) 

3U 

It can be seen frcaa {2.5.7) - (2.5.12) that 

PnWl = - P„(li)l' 

after long calculations. Further ^from (2.5.12) we get that 
for all u 

i(p) = lim n“^ E {— log p_{y)l^ 
n-vco 3p ^ 

= 2-. (2.5.13) 

We shall now show that the conditions (H3) and {H5) are 
satisfied by this process. Clearly, since o 5 r < k, from 
(2.5.12) we get that 

— i 3'^ 

lim n — - log p_(p) = 0 

n-^oa ^ 

for fixed k, which in turn shows that 

lim lim n”-^ log p (y) = 0 
k-»^ H-+OB 3y^ 


(2.5.14) 



Furthermore, we have 


n 


-1 


3_ 


log 




R 


n 

C 2-1 
i=l 


Y^-Cx, 


-Cu) - n' 


3p 


log p^(p) 


- n“^ y ^ — loa p(x,. 
i£i3y - ^ 


(i-l) k+r+1 ' * * * '^ik+r^ 
...{2.5.15) 


Let 




n 


Then 


1EECy“*<X^-Cp) 2^31 


= n“^ C^\y^‘ *1 jE{(Xj^-Cp} (X^^-Cp)}| 

+ n~^ }'E{Xj^-Cp)^j . 

From (2.5.10) and (2.5.11) we have 

lYn*^ ^ ® (d-p)^ (1+p)}"^ (2.5.16) 

For all i = l,2,...,n. Therefore, 

jE{CYjJ*{Xn-Cp) 2^3 {< 64(1 p|+l+p2) {n{l-p^) (l+p)^}~^ 

< 192{n(l-p^) (l+p)^}"^ . 


Further 

Var(CX„ Y^*) ^ C^{y^')^ (l+p^) 
“ n ^ 
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< 64{(l -p)^ (l+p)2}”^ (l+p^) 

< 128 {{l-p}^- (l+p)2}-l , 

so that Var(CX^ iiniformly bounded in n. Hence, 

by Leirana 2.2.5, we get that 

1 ^ 

{a.s)lini C n“-^ I y^’(X.-Cy) =0. (2.5.17) 

n-Ko i=l n X 

Now, for any s > 0 

tljjlog Pj-<U){ > el < (ne)-^ Pj.(Pll^ 

= (ne)"^Ir-2ptj. (l-p^)(l-p)] 

by (2.5.12) . Since 0 < r < k, the right hand side tends to 
zero as n -»■«>. Therefore, 

lim {p)lini n"^ — log p (v) = 0. (2.5.18) 

Since { — log ’ i > 1} is a 

stationary 1 -dependent sequence of random variables ^ it is 
ergodic and hence , by ergodic theorem, the third tem on the 
right hand side of (2.5.15) converges almost surely to zero 
as n °°, for every fixed k. Hence by (2.5,15), (2.5.17) 
and (2.5.18) we get that 

lim (p)limn"^ — log <|i^(y) =0, (2.5.19) 

k"^°“ 3ii 

which is (ii) of condition (H3) . Now, consider 
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- , V k 

+ c2 n"^2-I I yj* • 

i=±l j = l ^ 

= 2p {n{l-p) - (l-p^)t^ 

-v(l-p^)tj^3 (2,5.21) 

by {2.5.12). Since this expression is independent of y, 

sup n“^ “.log <j)j^{y’) = n“^ i_ log (j), {y) 

U'eVj^Cy) 3y^ ^ 9y^ ^ 

for any neighbourhood V^(y) of y. By (2.5.21) , for fixed k, 
wa have 

lim n-1 log > (y) = -2p{k(l-p)}~^ (l-p^)tT-, 

n-Ko 9y ^ ^ 

so that 

— T ^ ^ 

lim lim n j- log 4) (y) = 0. 
k-x» n-*^ 3y ^ 

Hence 

sup — - log 4'. (y)l = 0, 

y‘eVj^{y) 3y^ 


lim lim {n 

k-H» n-*^ 


(2.5.22) 
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For any neighbourhood of y. Thus (2.3.4) is satisfied. 

32 

Since log Pn^^^ independent of , <H5) will be 

satisfied with . ,x^) = 0. Thus all the conditions of 

Section 2.3 are satisfied and the BILE of U is a consistent 
estimator by Theorem 2.4.1 and first-order efficient by 
Theorem 2.4.2, 

Now, we shall prove the asymptotic noinnality of the 
MLE of ]i. Let be the true value of the parameter p. 
Then 

^n = ^log (2.5.23) 

3|i 

By (2.4.37), in order to find the asymptotic distribution of 
Zjj ~ enough if we find the 

asymptotic distribution of We have 

^n * Pn<t^o^ * (2.5.24) 

Consider 

'’"'"o’ “ ^n' 

= C S^ (n s2,-l/2 2 j^^i.(X.-Cy„) 

where 

n 

4 “'e yi- {Xi-Cv^)!^ 

= n-lE[C S^log p^(m„) 12 
E[|^log 


■1 
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= [%~2pt^ d-p"') {n(l-p)}"^] (2.5.25) 

by (2.5,12). Let 
n 

n x,-' 

where 

«l-CPo> • 

Then it is clear that, 

(i> the random variables _ are 1-dependent for fixed n, 

-t- gxl 

(ii) E(Y. ) = 0 for all i and n , and 

i,n 

n 2 

(iii) limsup ^ E{Y. ) 

n-^ i=l 

n 

= limsup s“^ limsup n“^ ^ (y^*)^ E(Xj^-CPq)^ 

n->°> n-x“ i=l ^ 

= (l+p^) (l-p)^ limsup n”^ I • 

i=l ^ 

Since E(Xi~Cy_) " =‘l+p^, = (1-p)^ and lim s^ = 1 by 

(2.5.25). Hence by (2,5.16) , 

limsup f E{Y^ )^ < 54(l+p2) (l-p)^{(l-p)^(l-p^)}“2 

n-xo 1=1 ^ 

= 64(l+p^) {(l~p) (l-p^)}"^ < «. 

(iv) Var(C ) = 1 for all n, 

n 

and 

(v) if }? j is the distribution function of Y. f then for 

ni i,n 

every e > 0, 
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n 

lim I / dF . (x) < lim e“^ | / jxp dF . (x) 

i=l lxi>e n^co 4lji|>e • 


< e“^ lim I E{Yj^ 


n 


n-*<» i=l 


,n< 


< e"^ lim s‘^ I |y^‘)^ Elx^-Cy 1 

n-Ko i=l “ 


= e~^ (2/2 A) lim f |y^*|^ 

i=l ^ 


n-x» 


since EjXi-CiiQj^ = 2/2/it and lim = 1. Hence, by (2.5.16), 




n 


lim T / X dF .(x) 
n-4H» 1=1 j X 1 >e 


<a£^ (l-p)^ (2/2/Tr) lim 


(2.5,26) 


n-»-«> 


where a = 512 {(1-p)® (1-p^)^} Since the right side of 

(2.5.26) tends to zero as n ^ 


n 


lim / x^ = 0 

n-H» i=i jx]>£ 

for every e > 0. 


(2.5.27) 


Thus all the conditions of Lemma 2.2.6 are satisfied. 
Hence converges in distribution to normal with mean zero 
and variance enttfe. Since s^ tends to'X as n ~ , we get that 
log converges to noinnal with mean zero and 

variance Ifflwo. Therefore , by (2.5.24), since {n/I^^ (U q) 
tends to 1, we get that 
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’'n = P„(Uo) 

converges to nomal with mecin zero and variance Since 

the asymptotic distribution of 

1 /'2 

same as that of Y„ , we get that {I (u- ) } ^ (u -p ) converges 

n ^ no no 

to normal with mean zero and variance on<e. This proves 
asymptotic normality of the !ILE of p. 

2,6 Conclusions . 

In this chapter, we proved wecik consistency and 
first-order efficiency of a MLE for ra-dependent sequences 
of random variables, under some regularity conditions. 

From definition 2.2.1, it follows that, when m is equal to 
zero, m-dependence reduces to independence of the random 
variables. Then clearly (2.3.1) becomes 

n 

p(Xj^, . . , .,Xj^ p{x^;9)} <{ij^(x^ ,. . .,Xjj;e) (2.6.1) 

so that (Xj^, . . . ,x^; 6) =1, for all k. It can easily be 
seen that all the conditions of Section 2,3 will be 
satisfied and hence Theorems 2,4.1 and 2.4.2 are valid 
in this case. In fact, tlie strong consistency and 
asymptotic normality of MLE hold in this case. 



CHAPTER III 


MAXIMUt-i LIKELIHO O D ESTIMATICM TOR 
STATIOI'IARY 9 - M IXING PROCESSES 

3 . 1 Int roduct ion . 

In recent years, several authors investigated the 
properties of stochastic processes satisfying certain 
"mixing conditions" {for evarpia , see Ibragimov {20} , 

Serf ling [46] and the references there) , Roughly speaking, 
any mixing condition says that the dependence between the 
random variables is weaker the farther they are apart , or 
else the dependence between the end of the process zuid its 
beginning is weak. 

In this chapter, we establish weak consistency, 
asymptotic normality and first-order efficiency of an 
approximate maximum likelihood estimator {AMLE) \idien the 
observations on which the AMLE is based are from a 
stationary stochastic process satisfying a ^^ixing 
condition. Definitions of AMLE and ^'-^nixing process are 
given in the following sections. 

Under the set of conditions presented in this 
chapter, the proof of weak consistency and asymptotic 
normality of AMLE depends on weak law of large nmobers 
and central limit theorem for double seqT:^nces of random 
variables respectively. Rosdh [40] proved a central 
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limit theorem for doxxble sequences of random varicifales, 
under some general assumptions. We obtain a particular 
case of this central limit theorem for stationaary (^-mixing 
random variables in Section 3.4. In Section 3.3, we use 
Khintchine's method to obtain weak law of large numbers 
for double sequences of random variables. Uie proofs of 
weak consistency, asymptotic noriuality and first-order 
efficiency of an MiLE are given in Section 3.6. Section 3.7 
includes an example which satisfies the regularity conditions 
given in Section 3.5. 

3.2 Preliminary Results : 

In this section, we give some definitions and lemmas 
which are needed to prove the main results of the chapter. 

Let {Xj^, n>l}bea sequence of random variables 
defined on a probability space. Letift^ denote the u-field 
generated by events of the form ^ (X^^ ,. . . e E}, 

where a-1 < i, < ... < i. < b+1 and E is a k-diroensicmal 
Borel set. 

Definition 3.2.1 ; A sequence {X^^r n > 1} of randcm 
variables is said to be (|)-m.ixina if for each t{l it < *») 
and for each n{n > 1) , 

jp(Bjh^) - P{B)| < (|»^ (3.2.1) 

with probability one for all events B ?diere 

is a sequence of positive numbers tending to zero as n 
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Remark 3.2,1 ; Condition (3.2.1) is equivalent to 

jPCAQB) - P(A) P(B){ < P(A) (3.2.2) 

for every events A efn? and B eyrt^ . Proof of this result 

J- t+n 

can be found in Ibragimov [20] . 

Remark 3,2,2 ; If {X^r n >1} is a m-dependent sequence of 
random variables (see Definition 2.2.1), then (3.2.1) will 
be satisfied for some sequence { 4 )^} of positive numbers 
such that <l>n “ ^ n > m. 'Thus m-dependence is a 

particular case of ({> -mixing eind hence the results of this 
chapter will be true for any m-dependent process satisfying 
the assumptions given in the next section. 

Now, we state and prove some lemmas. Proofs of these 
lemmas can be foxind in the references given below. 

Lenana 3.2,1 ; (Ibragimov [20]). Let {X^, j > 1} be a 
stationary (})-mixing process. Let the random variable f be 
measureible with respect toTO^ and f ^ be obtained by setting 
fj = f (Xj ,^. . . ^) . Ihen the process j > 1 } is also 

stationary and <{)^-raixing ,where 

{ 1 for n < k 

(3.2.3) 

^(n-k) for n > k 

Lemma 3.2,2 ; (Serfling [46]). Let {Xj^} satisfy condition 
(3.2.1) . Let C be any random variable neastirafale with 
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respect such that Ej^[P< co for some p > 1. If 

1 £ ot < P# then 

EjECgjmf) - E(C) {“ 

< {Ej (3.2.4) 

V7here p"^ + q~^ = 1. 

Lemma 3.2.3 ; (Ibragimov [20], Billingsley [10]). For a 
sequence of random variables , if ^ is measurable and 
ri is iaeastxrableTri:^_j,^ (n > 0) , then Ej?lP< «, 
and p~^ + q”^ = 1 imply 

lE{?n> ” E(a E(n)l< 2 ({,^/P {E|?p}^/^ {E|niq}^/'5. 

... (3.2.5) 

Lemma 3.2 .4 ; (Cramer [14]). Let ^ 2 / t>® a sequence of 

random variables, with the distribution functions (d.f.) 
Fi,F 2 ,... . Suppose that Fj^{x) tends to a d.f. F{x) as 
n Let , Ti 2 » ... be another sequence of random 

variables, and suppose that converges in probability 
to a constant C. Let 

^ \ = En/%- 

Then the d.f. of tends to F(X'-C) . Fxirther if C > 0, the 
d.f. of Y„ tends to F(x/C) while that of 2 tends to 


F{Cx) . 
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Lemma 3.2.5 ; For any random variables X Y and Z, 

ElE(Y|X) -E(Y)) < ElECZjX) - E{Z) | +2E{Y-Z}. 

Proof ; We have 

E|E{YjX) -E{Y)1 < E|e{Y-Z| X) - E (Y-Z) | + e} E (Z } X) - E(Z) f 

< 2EjY-Zi + EjECzjX) -E{Z)|. 

Before we proceed further, we shall say a few words 
about the Levy metric. Full treatment of these concepts can 
be found in Gnandenko cind Kolmogorov tl7] . 

Let D be the metric space of all probability 
distributions on the real line, with the i^^y distance 
L ( . , . ) , that is , 

L{F,G) = inf(hjF(x~h) - h < G(x) < F{x+h)+h, all x) , 

...(3.2.6) 

Then D is a complete space and convergence in the sense of 
the metric is equivalent to convergence in distribution. 15xe 
following lemma is probably not new. 

Lemma 3.2.5 ; Let X and Y be any two random variables. Then 

. ■' /{2+6} 

L(£,CX+y) , X{X) ) < E{jYP (3.2.7) 

for any 6> 0 where £{X> denotes the distribution of X. 



Proof : yie. have 


P{X+Yix) . • 

< P({y{ < h , X<x+h) + p(}y|>ii) 

i Fx(x+h) + P({Yl>h) " . C3.2.85 


By siibstituting X+Y , -Y and x-h in the place of X, Y and 
X in (3.2.8) respectively, we get 


Fj^Cx-h) < Fjj^yCx) + P(|y}>h). (3.2.9) 

From (3.2,8) and (3.2.9) we get 

Fx{x-h) - P(jYl>h) < Fjj^y{x) < Fjj(xth) + P{iyj>h) 

... (3.2.10) 

for all X. Hence, from (3.2.6) it follows that 

L(X(X+Y) , X(X)) <h (3.2.11) 

if h satisfies h > P(jyf>h). An application of C3iebyshev*s 
inequality yields that this is fulfilled if 


for any 6 > 0. Hence the lemma follows from (3.2.11). 

The following three lemmas are extensions of 
Theorem 2.3, I^mma 2.1 and Theorem 3.1 of Serf ling £46} to 
double sequences of random verifies . !ISie p3x>of s are similar 
to the respective results in Se||fling (46] . 
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Lemna 3.2.7 ; Let. {x_ •, i = g ,n>l}bea double 

se<|uence of random variables satisfying the following 

conditions . 


(i) °° as n ->■ «>. 

(ii) The random variables in the nth row ace stationary 

(n ^ 

emd 4" ^ ^ -mixing , where 

{3.2.12) 

for all n >1; 1 < k < 8 . 

- - n 

(iiil E{X .) = 0 for ail n and i. 
n /X 

(iv) lim e(X ss m <«» for some 6 such that 0 < 5 < 1. 

n-Hn r*" ~ 

(v) 4'v. ® ® (n“®) f for some 6>1 + — where 6 is as in (iv) . 

” 5 

Then, there exists positive constants C, 2. and e such that 


EjE |S ^ } - E(t 2 ) j 

aSnrA “8n/A 


< c{a2 + 


atAB 


for all n , where = b;;V2 x„ _i, 


a 


and 0 < a < a+A < 1. 


Proof ; By assumption (ii) and Minkowski inequality, we 
have , for all e > 0 , 




(3.2.13) 
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s sequence of positive integers such that 
Yn = 0 < ^ < 1* 

S ^l®«5en+Y„,ilSa,n) ' >i 


+2E| - "^aBn+YiifAl 


(3.2.14) 


by Lemma 3.2.5. By Lemma 3.2.2; assumptions (i) and (ii) 
the right side of (3.2.14) is 

< 2{(f>^^^} -fEt T [2+e ,2/(2+e) 

- ^ ^^'^a8n+Yn/Al ^ 

n 

■*■ '*' ’'aSn+Yn'*^ I ’'’oSn >A " ’’aSn+Yn-®^^ 


< 2{* {E |s" |2+e,2/(2+=:) 

Ijj ^ f« 


2®<l''a6n,A ’^c<S„.Y„,aI ^\.pr VA,nl>- 

,1/2 




where R = 3 
o/H n 


y X 2 . The right side of (3.2.15) 
i=aB^-!-l ^ /• 


xs 


< 2{* tElS, |2+^)2/<2-^"> 


'n 


A,n’ 


* ^"^l’'aB„,A + ’^cS .y„,aI 


2„l/2 




n n' 
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< 2U {E|s, 

‘ A,n* 

by stationarity and C^. -inequality {Loeve [28], 
Therefore, 

®a,n) -E(t26„,4)| 


1 2{^ je/{2+e)^ ^.g j2+ej2/(2+e) 

Yjj ' 




Now consider 

Ag 

^n 




ABjj A8j^ 


< E(!^^;^) + 2 I |EfXn,l !^,j-i+l)U 


j=i+l 

A8n A$n 

< AE{x2 ) + 26^^ I I ( E(^ ) (3,5|a7} 

' x=l 3=1+1 J ' 

by Lemma 3.2.3 and assumptions (ii)and (iii) . Therefore* 

by (3.2.12) we get 

Aa, 


E[a 


- 1/2 


9 -> -1 ^ 1/2 ? 

I X .] < AE(X; ) +20 I I E(2r ) 

" n,l n i£3_ « *h,l 


n 


A0_ 00 


■8. >1/2 


= AE(x2 ,) + 20 E(X^ ,) I I {0(k (3.2.18) 

^ » A i=l k=l 

by assumption (v) . That is , 
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-^>2 2 


(3,2.19> 


whare 


^ {OCk”®)}^"^ < 03 , 

Iv^l 


(3.2.20) 


Note that 0 > 2 since 5 <.l. Now, by (3.2.13), (3.2.16) and 
(3.2.19) we get 




) I 


n 


£ 2{* }"/<2«){ 

+ 8[{ a+2M2)A E(X^^j^)){(1+2H2) YjjS'^ A> 

Then, since and O < X < l,by assumption (v) , the 

right side is 

2+e,2/(2+e) 


,-Xee/( 2 +e) 2 ^ 2 Y/( 2 +e) 

3 ^n ^ ®n 


< 2M, - A- e;-' '- " > 


+ 8(1+2M2) g^(i-X)/2 E(X^^^) 

where is a constant independent of n and 2 y = 2+e, 
OSierefore, we have, by Holder's inequality. 


.a1s„ J - E(T 




“®n 


) 


+ 8(1+2M2) 


(3.2.21) 
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Since 0 > 1 + 2/5, there exists an e such that 0 < e < 6 
and 6 > 1 + 2/e, that is , 9t: > 2 y. For this s, if we 
choose X = 3Y/(ee+Y) , then 0 < X < 1 and {3. 2. 21) will 
imply that 


EiT^ fS^ „) - EiT^- , ) I 


< 2«3 8-(«'^-2r)/(20e«v) ,2 

+ 8(1«M2) 

= Cb/ (4^+4^^^) {EiX^ (3.2.2: 

where 


C = 2 H 3 + 8 ( 1 + 2 M 2 ) , ( 3 . 2 . 23 ) 

£ = { 0 e- 2 Y)/{ 2 ee+ 2 Y) , ( 3 , 2 . 24 ) 

Y = 1 + c /2 , ( 3 . 2 . 25 ) 

and 

0 <e< 5 . ( 3 . 2 . 26 ) 

Hence from (3,2.22) - (3.2.26) the lemma follov/s. 


Lemma 3.2.8 ; Under the assumptions of the previous lemma, 
there exist positive constants and g such that for 

eveiY n. 




n 




=a,n> 


% 



2IIS7 
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Proof; Let Y „ . = EfT"^ .!s ) - EfT^„ 

aBj^r^ agj/* a,n^ 


By 


Cj.“inequality, assumption (ii) implies that 


= E|S, |2+« 

' A/H* 


i ^2+fi gl+6/2 gj^^|2+5^ (3.2.27) 


The last step follows by Minkowski inequality. Let $ be 
such that 0 < g < ZS/(2+2z+S} , where JE. is as in (3.2.24). 

a 

. .a1 


Let A denote the event { } Y _ , | < r } and AJ* its 
n ' aB„,A' - n Ti 

complement , where 


= = 6;’‘{eK 


1+a 


(3.2.28) 


Taking the expectation of j Y^^ a i ^ ^uid 


aBn / A 


separately , it follows that 

ElY ll+S ^ _-l -|„ I . _(«-2B)/2S , , 1+4/2 


(3.2.29) 


By (3.2.22) , we have 


r“^ eIy - J < 1 . 

n ' aBjj ^ ‘ - 

The second term on the right side of (3.2.29) is 


(3.2.30) 


^ ^(S-2i)/2e ^1+6/2 ^2+5 jl+S/2 , 


2+5 


(3.2.31) 


by (3.2.27) . Since (5-2a)/{2a) > (2+5)/(2X) , 0 < e < 5 
and 0 < A <1, by (3.2.28), we get 
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{6“23)/2e 


n 


< 0"^ 1+^/2) {Ejj^ ^j2+6j2 {5-2B)/(2+5)20 

...C3.2.32) 

where is a positive constant which depends on 0. Bierefore, 
from {3. 2. 29) -(3. 2. 32) , we get 




< 1 + n 

- 0 


(3.2.33) 


for n > by ass^jjnption (iv) of Lemm 3.2.7. Let 


d = max {E|X^^^|2+«, e|X2^j|"°,„.,e|Xj, H) 

Ihen, 


1 2+5 


, 2+6 


El’^aS < 1 + C„ 21*^2 ^d+Bj/B- 

n » ^ 


for all n, where 

1 + 6/2 n + 0)/0 

C' = 1 + C„ 2 ^l-*-rp;/P 

6 B 


(3.2.34) 


(3.2.35) 


Hence the lemma follows. 


Remark 3.2.1; Note that the estimate in the lemma holds 
for all 0 such that 0 < 0 < Jl6/(2+2il+6) . 

Lemma 3.2.9 ; If the assumptions of Lemma 3.2.7 are satisfied 
then there exists a x such that 0 < x < 5 and Lx AucJn that 

Ef I X . {2+'^< 01+V2 L 
' n,i' - n ^ 

for every n > N.^. 
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Proof ; Since 2 £5/ (2+2 £+5) > 0 in Lemma 3.2.8 , there exists 
a positive number x such that 

t/2 < £'6 (2+2£+(S)“^ . (3,2.36) 


We take 8 in Lemma 3.2.8 to be t/2. Without loss of 
generality we can assume that 


8 < minc|- , p 


so that 

0 < T < min[l, 5] . 


(3.2.37) 


en. 


Let in - [ , the greatest integer < define 


m 


2m 


Rp, TO = y X . and S . = T X . . 

n,i n,m ^4+3^ n,x 

Since T< 1 , we have, by C^. -inequal ity ^ that 

1+T 


+ |Sn • 


(3.2.38) 


Now, letting h E{S^ { R ) - E(S^ ), we have for 

j-xv^w, iiy “n, m ' n,m‘ n ,m^ n,m 


0 < s < 2 




s/2 


i 
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by Holder’s inequality, 111011 




“ ‘ n fTn ' ' n ,111 


by Cj. -inequal ity . Hence ^ for r+s = 2+t and 0 < s < 2, 

* A^^|1«/2js/(2+t)j 

by Holder's inequality. Therefore, 


E 


. /f,, >s/(2+t), 

^ t/2^ ^ 


by stationarity ^ {3.2,19) with A = m 3”^ and Lemma 3.2.8 


n 


By assumption (iv) of Lemma 3,2.7 for sufficiently large 


n, say n > N^, we get 




where C_ = {C' }s/(2+t) ^ Therefore, for n > N 


2,t 

(N^ > 


T/2 


1 ' 


eK ris,, ,,1® < C.{e1r |2+'^}r/(2+T) ^s/2 (3.2.40) 

‘ hi,mi •‘^n - 2 ‘ n ^ 

where C 2 depends on s. Define for positive integers h. 




(1+t/2> 


n 


e! I X 1 

i=l 


2+t 


(3.2.41) 
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which, is finite by assumption (iv) of I^mma 3.2.7 since 
T < 5. Then from (3,2.38), there exists a constant 
such that for large n (say) n > N’{N' > N^) 


ElRn,m + S 




2+^ < 


where 


+ C4E2{e1r., j2+Tji/(2+T) ^{l+x}/2 


pfn 


+ {ElR^^^j2+'^}2/^2+T) ^t/2 






m,n 


^ C.[2{A„^„ ,l+V2jl/(2«) ^(l+x)/2 
+ 2{iS„,„ „l+V2j{l+x)/(2+T)y/2 

+ n,l+^/2j2/(2+T) ^t/2 

ra ^1 +t/2.V(2+t) 

+ ^\i,n 

(3.2.42) 


I+t/2 

^ ^ :n^2+g(ii.n ':.) 1 


g(Z) = C^[2Z' 


.(1+t)/(2+t) ^ ^^-l/(2+T) ^ ^-T/(2+T) ^-2/i2+T)^ 


Since g(Z) -^0, as Z there exists Zq > 1 such that 

{ 2 +g(Z) S 21/2 . ,-l (3.2.43, 

for Z > Z^. Define 



57 


a, „ = inax(A, M) 

li,n O' 


C3.2.45} 


where M is as defined in assumption (iv) of Lemma 3.2.7. 
Since Z(2+g(Z)) is a non -decreasing ftmction of Z, we have 


Hence from (3.2.42) and (3.2.43) it follows that 

+ Sn . mP *'' < »i^,n((2^/2_2-l,2+T _ 

Since 2m < ^ we obtain, by Minkov/ski inequality that 

* {Eix, , 

n 

< ( 2 ^/^ - 2 - 1 , . 

“* ni^n o m^n 

< -1''= - 2-1 * ™-V2, . 


(2-' 


- 1/2 

Choose n sufficiently large, say n > so that m ' < Z^ 

"Then for n > max(N‘, N 2 ) 

El l" X < ( 2 m, 1'"!^^ a 

n , X ' ~ m , n 


< a 

" n m n • 


(3.2.45) 


It follows from (3.2.45) and the definition of Ag ^ th^t 
\n - ^m,n' ^ ^ isax(M', Nj) • Let H = ^rmxin* ,^ 2 }' 
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Then 


^ ,n ^ such that 8^ > N. Suppose 


n is such that N < 6„ < 2N , Then - < m < N and hence 

n - 2 - 


n 


^ n r- ^ ^Tr» r' ^ HlaX f ^ ^ » f } 

^n'^' " " l,,n^ N,n 


(3,2.46) 

How, we suppose that if n is. such that 2^“^H <8 < 2^ H, 


n - 


tnen 


i < max (a, , , , , ,a ) 

Pjj ,n “ J. ,n N ,n 


(3.2.47) 


Ir *k'*4“X 

and consider all those n’s for which 2 N < 8 < 2 N. 

n - 

Then 2^”^ N < m < 2^ N so that, frcrni (3.2.47) it follows 


that 


m 


< max (a-, „ , — , 

,n ” i,n ^ ^ N ,n • 


Hence by (3,2.46), we get 


Bn,n - p".---' ^N,n> 


for all n satisfying > N. Therefore, for 8jj > N 

0, 


I V l2+T nl+'^/2 , 

I ^n niax(a 


i=l 


n 


l,n'* ,n^ 




for some k such that 1 < k < H. Since t < for all n 


such that Sjj > N 


Ai =k-<^+^/2) 
,n ^ 


JV 

iL 


2+T 






:2+T 





by Minkowski! inequality and stationarity . Hence 
< M 


A- 


k ^ 


by assumption {iv) of Lemma 3«2,7. Let 
, l+T/2 ’ 

L = max (MN , , M) . 

Then L is finite for all n such that 6 > N and 

n 

= max (A- ^ , 2 M) < L~ . 
kpi K;,n o' - z 


Hence 




{3.2.48) 


for all n such that $^ > N. We note here that this N may 


depend on t. From (3.2.48) it follows that 

= 1 .L \ i 


(3.2.49) 


for all greater than some N. (This N need not be same as 
the above N) . Thus the lemma follows. 


Definition 3.2.2; A function g(x) is said to be uniformly 
integrable with respect to a family of distribution functions 
{Fj^, n > 1}, if 

lim limsup / ^q(x)i dFj^(x) =0. (3.2.50) 

C->“ n-^ |x|>C 

Definition 3.2.3 ; A family H of distribution functions is 

said to be relatively compact if every sequence of elements 

of E contains a convergent subsequence, co-nWi9i«3 & a diA^ rxihdiDyx 



Leimna 3 . 2 , 1 0 ? (Loeve [28]). if for a given > 0, [xj^® 

is uniformly integrable with respect to the family 
{Fj^, n > 1>, then the sequence n > 1} is relatively 

compact = 

Proofs For proof, we refer to Loeve [23], p.l84. 

3.3 Weak Law of Large Numbers for Double Sequences of 
Random Variables ; 

Let {y , i = 1,2, .,.,6 ,n > 1} be a double sequenee 

X H 

sequence of random variables. We shall assxmie that " 

as n We first introduce some notation, related to the 

double sequence , which will be used throughout the later 
sections . 

T/jhen we put a non- integer X in a position where there 
should be an integer, then X is interpreted as its integral 
part. Empty summation yields zero. £(X) stands for the 
distribution of the random variable X and is also used to 
denote the distribution function F^{x) = P(X < x). E(xj^) 
stands for the conditional mean of X, given the u^field^. 

We usually write E(xjy) instead of E(xl^(Y)), ^diereJBCX’) 
is the a-field generated by the random variable Y. N(u,o > 
stands for the normal distribution with mean y and variance 
■y 

. 'VJb write 


2 


2 
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= Y T + 
n ,1 

V- 4. 

*•“ ^ 1 « < 1 r 

n 

^ A 

a, A 

a+Ag_ 

= I 

i=a4-l 

Y • 0 

n,x ^ ^ 

1 

f < A < 1. - 


and 




0 < a < a+A < 1. 


We denote by G and G^ „ 

°>0L a,n o ,n 

the distribution which has its 

In fact, S = and 

o,a 


5^, where 5^ stands for 
entire mass at the point \i. 


G 

airn 



) . 


Now, we state and prove the weak law of large numbers 
for double sequences of random variables. The proof is 
similar to the proof of central limit theorem obtained by 
Rosen [401 . 

Theorem 3.3.1 ; Let {y . i = 1 ,2 , . . . ,B- ? n > 1} be a 
dOToble sequence of random variables which satisfies the 
following conditions ; 

(i) The random variables in the same row are defined 
on the same probability space. 


(ii) lim e( S „ “ S < xi^~ X) > Q < y < a< \ for 

a^n Y /I - » _ 

some ,6>0 and for some function x(s) which is bounded on 
0 <■ s < 1 and tends to zero as s tends to zero. 


(iii) lim lim EjE(T^D =0# 0 < a < 1. 

A-h< 5 n-^ ^ 

(iv) lim A“^ Tim / x dG^ . (x) = 0, 0 < a < 1 for every 

A-^+o n-x” I xl >e 

e > 0 . 
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I 

i 


Then for 0 < a < 1, 


(p)lim S = 0 

n-><» ^ 


We assimie that 


lim „ = G , 
a^n a 


(3.3.1) 


(3.3.2) 


for 0 < a < 1 ^ and afterguards show that this condition is 
superfluous. The proof of the theorem d :pends on the following 
lemmas. The proofs of all these lemmas and the theorem are 
similar to those of the corresponding results of Eosen [40] . 


Lemma 3.3,1 ; Wien condition (ii) is satisfied, the family 
{G^ j^{x), n > 1} integrates |xj^ uniformly for 0 < e < 1+5 
and for fixed a. 



so that the lemma follows. 

Lemma 3 , 3 „ 2 g Under assumptions (ii) and (3.3.2) , the family 

{g (x) , 0 < a < 1} is continuous in the Levy netric. 

a _ - 
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proof o Let y < Then 


L{G,i^ G ) — lira L(G . G ) 


a y 


n-^-oo 


lira L(£(S „) ,X(S )} 


= lira L{X{X^+Yj^) , X(X^)) 


T1-K50 


where and Therefore by 

Lemma 3.2,6, vre get 


L(G , G ) 
a Y 


< 


{Tim EjY^l 
n-voo 


1+5j1/(2+5) 


< 


{x(ot“Y) J 


l/(2+5) 


by condition (ii) , Since -»• 0 as a-y ^ 0, the 

lemma follows . 


Lemma 3.3.3 ; If assumptions (ii) and (3.3.2) are satisfied, 
then the function (|)(t,^a) ? which is the characteristic 
function of G^, satisfies 

(a) 4)(t,a) is continuous for 0 s; a l -eo < t < «^ and 

(b) ^(t,a) exists and is continuous for 0 < ot < 1# 
ot 

-■» < t < “ . 

Proof ; Since the characteristic functions converge uniformly 
on every compact t-interval whenever the correspontog 
distribution functions converge, (a) follows from Le 



According to assumption (ii) and Patou's lemma, it 
holds that, for some 5 > 0, 

Jjxl^'^'^dG (X) <m /l^l^'^'^dG (X) 

^ n-^oo 

< sup x(ct) < 

0<a<l 

Thus -1- d) (t ,a) exists and 
at ' 

00 

^(t,a) = / e^^^ ix dG (x) . (3.3.4) 

at i«, 0^ 

As Gj^(x) , 0 < a < 1 has a uniformly bounded (1+5) th moment, 

this family integrates jx] uniformly (cf. Lemma 3.3.1). The 

continuity of 4(t,a) now follows from (3.3.4), and (b) is 
at 

proved. 

Lemma 3.3.4 : Under assumptions (ii) , the family 
{G„ „(. k ) , n > 1} is relatively compact for every fixed a, 
that is every sequence of elements in {G(j^jj(x), n > 1} 
contains a subsequence which converges. 

Proof ; By Lemma (3.3.1), |x|^, 0 < e < 1+5 is uniformly 
integrable. Hence the conclusion of the lemma follows from 
Lemma 3.2.10. 

Lemma 3.3.5 : If the double sequence of random variables 
satisfies assumptions (i) “■ (iii) and (3.3.2), then <l>(t,a), 
the characteristic function of G^ satisfies the differential 

equation 


— <J)(t,a) = 0 
3a 


(3.3.5) 
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for 0 < a < 1, ■=«> < t<“ and has boundary values 

1 

ci)(t,o) = lo 

Proof; Since G = 6 for all n, where 6 is the 

— — P i-u vj O' 

distribution which has its entire mass at the point zero, 

the assertion about the bound ry values follows from 

Lemma 3.3.3(a) . To derive the differential equation we 

first show that (3,3,5) is satisfied, if i_<j>(t,a) is 

+ 3a 

changed to the right derivative — a). We have 

3a 

—1 

— tj)(t,a) = lim A [(j){t,a+A) - (f>(t,a) ] 

3 a A"*"'to 

= lim A”^ lim E[exp(it S . . ) - exp(it „)] 

A-^'+o n-»x» ' 

= lim A“^ lim (3.3,6) 

A->-+o n-><» 

where 

= Elexp(it S^^^) {exp(it - 1}] . 

We will now prove that this limit value exists from which 

the existence of ^ — ({)(t,a) follows. By the elementary 

3a 

inequality , there exists a universal constant C such that 
1 H ( X ) I < C min ( [ X | , X^ ) 

for all real X, where H(X) = e^ -1-iX, it follows that 
Ia^I = lE[Gxp(it s^^j^){exp(itTjg^^^) - 1}3} 

= |E[exp(it {it t Hit 





< lit E{e |E{e“®«.n H(fl^ ^, } | 

n n' 

** Xl 




... (3.3.7) 

Let e>0- The second term on the right hand side of (3.3.7) 


= Cj min{ltx| ,t^x^} dG^^^{x) 

= C / inin{ltxj, t^x^} dc” .(x) 

lxl>E/|tt 

+ C / min{ltx| , t^x^} dG^ (x) 

|x|<e/|t| 




s c|t| / |x| dc" .(X) + c €|t| eIt^b jI 

|x|>e/|t| 

Therefore it follox'/s from (3.3,7) that 


lim a'"^ lim |a J < jtj lim a"^ lim E|E(T^ ,Al®a,n^f 

. I ^ ** A-i.4-rv n->co n 


A">*+o n-x» 


A«^+o n-x» 
-1 


+ c|t|lii« A'-" lim / „,>I'’'^,A<=‘> 

A->+o n-w lx|>e/|tj 


C elt| lim a"^ lim e|T^ 
/\.^+o n-x® 


..(3.3.8) 


The first term on the right side tends to zero by 
assvimption (iii) , the second term tends to zero by 
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assumption (i^) and the last term tends to zero since e is 
arbitrary. By (3,3,6) it follows that 

(tir0t)~0e {33 9 ) 

3 ot • • 

Since the right hand side in (3,3,9) is continuous 
is differentiable with respect to a and is equal to zero. 
Hence, the lemma follows. 

Lemma 3.3,6 ; The function 

(j)(t,a) = 1 (3.3.10) 

is the unique (complex-valued) solution in the domain 
0<a<l,““<t<oo of the differential equation (3,3.5), 
which satisfies the boundary value condition (f>(t,0) = 1 for 

< t < oo . 


Proof s The general solution of the differential equation 
is (t ,a) = g(t) + i h(t). By the boundary condition, we 
get g{t) = 1 and h(t) e 0. Since (3.3,5) is a linear 
homogeneous equation, the solution is unique. This completes 
the rproof of the lemma, 

Now, we give proof of Theorem 3,3.1. Frcmi Lemmas 3.3.5, 
3.3.6 and from the one -one correspondence between 
characteristic functions and distribution functions, it 
follows that 


lim 

n-x» 


^ a?n 


(3.3.11) 
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for 0 < a < 1, when assxjmptions (i) - (iii) and {3.3.2) are 

satisfied. Since the family {G^^Cx) , 9 < a < 1} is continuous 

by Lemma 3.3.2, we get that (3.3.1) holds for a = 1. Next 

we show that (3.3.2) is actually superfluous, by contradiction. 

Thus we assume that the double sequence of random variables 

satisfies assumptions (i) - (iii) and that for soma a it 

o 

holds that 


lim X(S^ (3.3.12) 

n->oo O ^ 

According to Lemma 3.3.4, there is a subsequence {n^} such 
that 

lim X (S ) = A (3.3.13) 

y-^ “o '"y 

where 

A ?£ 6 . (3.3.14) 

o 


Again , from Lemma 3.3.4, 
subsequence {n^} of 
1 im G = G 


it follows that we can pick a new 
such that 

(3.3.15) 


for all rational a e [0, 11 fay a diagonal procedure. As in 
Lemma 3.3.2 , it follows that this family {G^, a rational 
belonging to [0, 1] } is uniformly continuous in the Levy 
metric. Thus we can extend it by continuity to be defined 
for all a e [0, 1] . Then we get that 


V-Kjo y 


(3.3.16) 
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for 0 < a < 1, because for any rational number r > a 


/ V ^ ^r> + G- ^ ) + U 


;^v 


r , n^''^r^ ' 
... (3.3.17) 


Ijemma 3.3.2 and (i) vield that 


n . n > < { X < r-a ) } ^ 


r,n^ 


(3.3.18) 


Thus, by letting v ^ ™ in (3.3.17), we obtain 


™ ^ {x(r-a)} 


1/(2+6) 


by (3.3.15). Thus (3.3.16) follows. We note that (i) - (iii) 
are unchanged when we restrict to a siibsequence . Since 
(3.3.16) is the same as (3.3.2), from (3.3.11) it follows 
that 


G — lim G — 6 . 
a , a,n o 

V-»<o V 


(3.3.19) 


Thus we get 


A = 6, 


which is a controdiction. Thus, under assumptions (i)--(iii). 


lim f,(S ) = 5 


(3.3.20) 


for 0 < a < 1. Therefore, 


(p)lim S „ = 0 

n-K» 


for 0 < a < 1 by Theorem lO.l.d of Loeve [28], p. 168 
which proves Theorem 3.3.1. 
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As an application of ^Theorem 3.3.1, we derive the 
following weak law of large numibers for dotible sequences, 
when the random variables in the same irow are stationary 
and <J)^ixing . 

Theorem 3^3. 2; Let i = 1,2,..., n > 1} be a double 

sequence of random variables satisfying the foilwing 
conditions : 

(a) The random variables in the nth row, that is, for 
fixed n, are stationary and <j> -mixing, where 

(3.3.21) 

for all n>i, lik<8 and 
“ n 

0, as k + ®. (3.3.22) 

(b) i^ “ ^ ^11 1 n. 

(c) im EjYjj = M, < «, for some 6 > 0. 

n-*o» ^ 

(d> There exists a sequence of positive integers 
n > 1} such that 

(i) Yjj = 

(ii) 0 as n 
^n 

Then 

(p)lim S, = 0. 

where ... + ^n,B^* 

Proof: We prove this theorem by showing that assumptions 
(i) - (iv) of Theorem 3.3.1 are satisfied. 
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LecBt^ 3.3.7 s If the assumptions (a) and (c) are satisfied, 
then the assumption (ii) of Theorem 3.3.1 will be satisfied. 


Proof : By Minkowski inequality , 

a^n 

= |Sa,n - = El J ^ 

i=YBjj+1 


1+5 


aS * 


< I I {EjY |1+5 j1/(1+5)j1+6 


1+5, 


1+5 


for some 6>0, by assimption (a). Therefore, 


^ n ” ®Y ni^^^ ^ im e|y„ 

n-K» ^ ^ 


= (a-Y)^^* \ 

by assumption (c) . Hence the lemma follows. 


(3.3.23) 


Leimaa 3.3.8 ; Under conditions (a) - (d) , assumption (iii) 
of Theorem 3.3.1 will hold. 


Proof ; For the Yj,^ , i>y Lemma 3.2.5, we have 


^ I ^ ''^6„ , J ,n I I ^ ' 

Consider the first term on the right hand side. By 
Lemma 3.2.2, (a) and (b) 
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'n 


C3-3.25) 


by stationarity. By taking y ■= 0 and a = A in the 
proof of Lemma 3.3.7, the right side of (3.3.25) is 

< }V(l+6) , jl+6,l/(l4-5) 

- Yn n 'nil' 

Taking limit as n-^ on both sides ,7 we get by (c) and (d) , 


= 0 . 


Further more, if R 


a 


, Y = .1 
' n x-i 


a$ 4 .Y 


<x$^+l 




e 1T^„ . - -J » EjR - - ^ 

aen+Yn»^ a, <a+^> 


(3.3.26) 


< 2E R 

a, Yj,' 

< 2{e|r„ „ |i+«)i/a+«) 




n 


< 2{y1+* , 


by Minkowski inequality. Assumptions (c) and (d) imply that 


®''^en+Yri'4 ' ‘ 


(3.3.27) 


From (3.3.24), (3.3.26) and (3.3.27) the required result 
follows * 

Leinma 3.3,9: If assmaptions (a) and (c) are satis f led i. ttieli 
assmuption (iv) of Ibeoreui 3,3.1 will be satisfied. 
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Proof: By <3.3.23), we have 


1+6 




1+5 


< Tm M, . 

n-^ ■*■ 

Hence 

lim a"^ lim * 0- 

A-^+o n'^ “Pn»^ 

Then the lemma follows by an argimient similar to that 
given in (3.3.3) 


Thus, it follows from Leimaa 3.3.7, Lenma 3.3.8 and 
assumption (a) , that all the conditions of Theorem 3.3.1 
are satisfied. Hence the conclusion of Theorem 3.3.2 
follows . 


3.4 Central Limit Theorem for Double Sequences of Random 
Variables ; 

In this section, we deduce the central limit theorem 
for stationary 4>"mixing processes from the following theorem 
of Rosen [40] . We use the same notation as in the previous 
section . 


Theorem 3.4.1 ; Suppose the double sequence 

{Y •, i=l,... ,8 ,3„ > 1} of random variables satisfies the 

following conditions . 

<C1) Hm ECSq n “ % n^^ - X(a-B)/ 0 < $ < a < 1 for 

Xi-*<a * * 

some function X(s) which is bounded on 0 < s < 1 and tends 
to zero as s tends to zero. 
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iC2) ’Hiere is a function p(a), continuous for 0 < a < 1, 
such that for 0 < a < 1, 


lim lim e{eCT^o . 
A-^-to n-*<=o '^n ' ^ 



Ap{a) 



(C3) There is a function (a) , continuous for 0 < a < 1» 
such that for 0 < a < 1 , 

lira ISiE.{E{(T^g js^ }- Aa^(a)} = 0 . 

A-»-+o n-^ n' ' 

(04) lim A lira / x2 dG^ *{x) =0, 0 < a< 1 for 

A -++0 n->«> {x|>e ' 

every e>0. 

Then 

lirailCS = N(0, t(ci))# 0 < a < 1 (3.4.1) 

n-+«> ' 



Proof: For proof of this theorem, we -srefer to Rosen [401 . 

The following theorem is a particular version of 
the above theor^i for stationary (jrraixing processes. 
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Leiama 3.4.1 ; If assumptions (i) , (ii) , (iv) and (v) are 
satisfied, then 


lim E (S 
n->«> 


a,n “ ^n^^ - X(a-e) , 0 < e < a < 1 


C3.4.6) 


for some function x(s) which is bounded on 0 < s < 1 and 
tends to zero as s -► 0. 


Proof : We have , 


ag 


n 


E(S „ - S „) = E( y Y .) 

<^Sn 

- I 


a 3 




E(y2 .) 

+ 2 ' 

n,i 

i= 


aSn 

r 

a3n 

L 

i-eB +1 
n 

1 

5=x+l 


n 


«^n 

I i^n 


n 



E{Y^ ,) 


n 

n,l 



a&n 

a3n 

+ 2 

I 

I 


i=e3ji+l 

j=i+l 


(n) ,1/2 


.. .(3.4.7) 

by Lemma 3.2.3 and assumption (i) and (ii) . Therefore, by 
(3.4.3) we get 

7-2 




“Sn “6n-i 

+ 2 E(Y^ ,) I I ’I'k 

^ i=e$jj+l ic=l 
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2 n 

1 {(a-e + 2 I I (3.4.8) 

i=e8j^+l k=l 

by assvimption (v) . ®iat is, 

^ (I+ 2 M 3 ) (3.4.9) 

where 

i {0(k"9)}^/^ = M < 00 , (3.4.10) 

k=l ^ 

by assiimption v) . Note that (iv) implies that 

lim 6^ ^ 1^ < “’* • Therefore, by taking limit as n 

on both sides of (3.4.9), the lemma follows. 


Lemma 3.4.2 ; If assumptions (i) , (ii) , (iv) and (v) are 
satisfied , then 

lim liM EjE(T”. .js„ )! = 0, 0 < a < 1. (3.4.11) 

A-»-+o 

Proof; By Lemma 3.2.2 and assumption (i) i-re have 


ElE(Y^,a6^4.ilSa,„)i e(y2 ) >1''" 


< 24. E(yJ 

1 ^ Jf 


by assumption (i) . Therefore , we get 

1/2 1 

<2{E(y 2 f 

- n /I ^^ 2 . ^ 


- 2M3{E(Y^_;^)}^''^ 


(3.4.12) 
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by assumption (v) and {3.4.i0). Therefore, 

By talcing limit supremum on both sides as n the 

lemma follows from assumption (iv) and the fact that 

S as n-*-w. 

. n 

Lemma 3.4.3 ; Under assiamptions (i) , (ii) , (iv) and (v) and 
(3,4,5) v(a) is non-decreasing and satisfies Lipschitz 
condition on 0 < a .<1, 


Proof; By assumption (ii) , we have 


E(S^ . ) = E{S^ ) + E(t'^q ,)^ + 2E(T^„ . .S ). 

a+A,n a0j^,A 

How, taking limits as n on both sides we get 


-n 


V 


(d+A) - v(a) = lim [E(t 53 + 2E(T^2 A • n' 

n-^ n'“ n^ ' 


• S,-. 3 

(3.4.13) 

by (3.4,5). Consider the second term on the right 
side of (3.4,13). Then, 


a3 A 


n 

(a+A) S 


n 


^n 


(3.4.14) 


where p^^ » +A and {y^} is a seqtienre of non-negttive 

integers such that 



79 


(i) as n 


{3.4.15) 


and (ii) * 


(3.4.16) 


Using Schwartz inequality for the first term and T. p.Tnm?> 3.2.3 
to the second tern of (3.4.14)^ by assumption (ii) , it 


follows that , 




2 , 1/2 


' ' i=aB +1 “ » 

n 

(n) 2 («+A)Sn 

+ EIS ). E( I Y 

^Yn i=Pn+l ' 

< {E(S^ )E(^|^ Y 

a,n n,i' ) 

- A(8 -Y-,} -5 1 /•> 

+ E(sl_^) E ( J“ yn,i> > 

...(3.4.17) 

by assumption (i) . By methods similar to those given in 
Lemma 3.4.1, we get 


4- (U2M3)A(B„-f„)E(Y2_;^)}l/2. 

Hence, by taking limit as n^ on both sides, the second 
term on the right hand side of (3.4.13) tends to zero, by 
assumptions (iv) , (v) , (3.4.5), (3.4.15) and (3.4.16). 
Therefore (3.4.13) reduces to 

v(a+A) - v(a) = lira E(T^g (3.4.18) 

n-><» ^n ' 

< lim {( 1 + 2 M 3 )A ^^^n,l^^ 

n-^ ' 
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by Leifflna 3.4.1. iSien by assumption (iv) , we get 

v(a+A) ” v(a) < {1+2M3)A , (3.4.19) 

where is a finite constant. From (3.4.18), it follows 
that v(a) is non -decreasing and the lemma follows frcm 
(3.4.19). Since ■^^(a) is non-decreasing and satisfies 
Lipschitz condition, v’(a), the derivative of v(a) with 
respect to a exists . 

Lemma 3.4.4 ; Under assmptions of the previous lemma 

lim a"^ limE{E{('I^g - Av’(a)[ = 0 (3.4.20) 

A-^+o n->*° ^n ' ' 

for 0 < ct <1, where v’(a) is the derivative of v(a) 

withjrrespect to ot. 

Proof: Consider, 

S«,nl 

+ E|E (T^- .)^ -Av’(a)i . (3.4.21) 

Since, by (3.4.19), 

V (a+A) - v(a) = liin ^^*^8 ^ 

ix-+<» n ^ 


< (1+2M3)AM3 
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An application of dominated convergence theorem 
shows that 


lim A"'- IS e|e(t" )^-Av'(c,)| = 


A**’+o n-Hxi ^^n f ^ 


(3.4.22) 

If we take in Lexwna 3.2.7, all the 

conditions of Lemma 3,2.7 will be satisfied and hence we 
obtain that 


A>^ iSe,n> .)' 


n r 




< C{A^+A^^^) e!y }2+e,2/(2+e) 

for all large n and £>0. Taking limits as n-^ on both 
sides by, assumption (iv) the right side tends to zero and 
hence the lemma follows from {3.4.21} and (3.4.22). 

Lemma 3,4.5 ; Under assumptions of the theorem 3.4.2, 


lim A"^ ° ' 0 < a < 1 (3.4.23) 

A-^o n-x=jx{>e a/ a 

for' every e>0. 


2 ,„n 


Proof glC-ien assumptions of the Theorem 3.4.2 are satisfied, 

if we take X„ • = Y_ %■ in Lemma 3.2.7, it follows 

11 , X n. “ 7 i 

that 

AB.. 

2+t ^ .l+T/2 


"n 


.1, ^n,il"" i 
1=1 ' 




(3.4.24) 


from (3.2.49) for all B > N. Therefore 

n 

A 0 n 

aBjj,A' i4i n ^i' 2 
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We have 


/ (x) < 

i|>e 


-T 


if, lx| 

X >£ 


2+t 


Krj, '*> 


< 


< 




,2+t 




~t^1+t/2 


M, 


(3.4.25) 


for all Bjj > N where T > 0. Hence the lenma follows. 

Proof of Theorem 3.4.2 : When the assumptions of the theorem 
and (3.4,5) are satisfied /it follows from Lemmas 3.4,1, 
3.4.2, 3.4.4 and 3.4.5 that all the four conditions of 
Theorem 3.4.1 are satisfied. Hence the theorem follows from 
Theorem 3.4.1. Now we will show that (3.4.5) is actually 
superfluous , Let 
aSn 

Vn(a) = E( ^ Yn,i^^ (3.4.26) 

for 0 < a < 1. The functions '^j^(ct) satisfy: 

v^(0) = 0, v^(l) = 1 (3.4.27) 


and 

VnCc+A) - - sl_^) 

< {2(Aa)^^^ + A) E{Y^ ,) (3.4.28) 

«« n II f -*« 


by stationarity , Schwartz inequality and arguments in 
Lemma 3.4.1. From (3.4.27) and (3. 4, 28), it follows that 
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■the sequence {Vj^(a), n > 1} contains a stib-sequence which 
converges, i,e., there exists {n^^} such that 

lim V (a) = v{a), 0 < a <1. (3.4.29) 

k 

(3.4,5) is satisfied for this stib— sequence. By argianents 
similar to those given in the case of weak law of large 
n^lmbers the theorem follows. 


3.5 Regularity Conditions ; 

Let n>l}bea sequence of stationary 

({(-mixing random variables. We denote by Pj^(x^,. . . ,Xj^;9) 
the n-dimensional joint density function which depends 
on a single unknown parameter 0. We have 

n 

log PjjCxj^, . . , ,x^;.e) = log p(x^; 0 |x^_^,... pc^) 


= log p(x^ . • j^e) 


n 


n 


+ 1 log 


i=)i^+i 


.. . (3.5.1) 


where{Jl } is a sequence of non-negative integers such that 
n 


C < CO, as n 


(3.5.2) 


We shall define the approximatB log-^likelihood 
function given the observations to be 
n 

log 5 (9) = I log p{xj^?9 -l' * * * '^i-^n^ ' 

i=J,j,+l “ 


(3,5.3) 
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Definit^n_3.5^; An estimator e 5 e_ (x, , . . . .x ) of the 

n X It 

unknown parameter 0, which maximizes log p (6) in a 

nGi^hbou^rliood of tno txno pB-irsiinotoir vqIus 0 - is S 3 .i< 3 . to 

o 

be an approximate maximum likelihood estimator (AMLE) of 0. 

Suppose the following regularity conditions are 
satisfied for all n. 

(Al> The parameter space 0 is an open interval on the 

real line and thr true parameter value 6 is an interior 

o 

point of 0. 

(A2) ^ log p (9) y i = 1,2 exist for all 9e0 and are 

30^ 

continuous in 9 with probability 1, where 
Pj^(9) = p(x^;9 jx^__^, 0 . . , x^) , 

For all Oe0 and for almost all ‘ '^n-1' 

(A3) EQ[|g. log p^{0) ! X 3 ^,...,x^_ 3 ^] =0 

(A4)(i) -Eq[|^ log p^{0) lxx,...,Xjj-ll 

= Eq[{|^ log p^(0)}^|xx 
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{ii) i( 6 ) 


-lim n“^ F fi 
n-»^ 


p(xi 


/X )] 

II 


1 ^ 2 

= lim n” I a (Q) 


n-H» 


k=l ^ 


< 00 


2 

;^ere 0 ^( 6 ) = EqI|- log Pj^Ce)!^ 


and i( 0 ) is positive, and 

(iii) for the sequence {£ } in ( 3 . 5 . 2 ) , 

n ' 

iir "'etfe = i(e). 

(A5) For every 6 e 0, there exists a neighbourhood V{e) of 6 
such that, for 0' e V(0) 


p^caM p^(e)i 


< |0“0’i G(Xj^ PC2 r . . . ,X^) 


1+5 


where 0 < G(x-, , . . , ,x„) and lim Ep, [G{X . . . . ,X ) ] < 

^ ^ 0 1 n 


n-H» 


for some 6 > 0, independent of S- 


(A 6 ) For all 0 e 0, 


lira Eglll log p “ 

n-«o ” 302 ^ 


for some 5 > 0 , independent of 6 . 


Hereafter all statements which involve convergence 
o.f random variables are valid with respect to the true 
probability measure Pg unless otherwise specified. Ife denote 
log Pjj( 0 ) by log Pj_ by ^^( 0 ) and 
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denote the derivatives of i{)j^(8) and 4*^ j^(9) 

respectively with respect to 8 and and ^(6) denote 

the second derivatives of (8) and i|i. (e) with respect 

i,n 

to 9 . 


3 .6 Asymptotic Properties of amlE; 

In this section, we obtain weaik consistency, 
asymptotic normality and first-order efficiency of AMLE. 


Theorem 3.6.1 ; When the conditions of Section 3.5 are 
satisfied, the likelihood equation has a root, which is 
consistent. 

Proof; Expanding Ui' ?6) = — log 0„(9) about the true value 
n 39 n 

8q of the parameter 8 , we get 


4 ^^(e) = 4^Meo) + + (e-e^) } 


n'“o' 
(3.6.1) 


where 9' = 0 + r(e--0_) and jrl <1. By (3.5.4) the 

o o 

right side reduces to 


Pn 

+ (9-9^) - 4^1 ^(60) 


{3.6.2) 


where tIj! (6), ijj" (0) and fi are as defined in Section 3.5. 
i,n 

It follows from assumption (A5) that there exists a 
neighbourhood V(9^) such that for 9’ e V(9 q), 
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_i) (3.6.3) 

n 

where 0 < GCx^, — '^i+s, snch that 

HE Eg jGCXj^,,.. ) j3-+« = M < 


n->«> o 


Therefore from (3.6.1) - (3.6.3) we get 

8,:^ '!'A<8> “ <6-8o> + (8-9o>' 

where j r^ | < 1 for all n and 


B = 

s"^ 

Sn 

I 

_(6 ), 

X /II o 

o 

n 

1=1 

B, = 

B-"- 

Bn 

1 

r- (e ) , 

1 

n 

i=l 

i,n o 

and 

-1 

^n 



®2 = 


iii 


(3,6.6) - (3.6.8) can be written as 

Sn 

B 


B. 


iL 

n, 1 


r. - (e^) 

l,n O 


y 

z _ . 

+ Eq 

r: {Q) 

Im o 

L 

i=l 

1 

H 

9o 


and 


®2 “ ^ ^n i G(x2^ ..,X£ ) 

^ i=l o n 


where 


’'n,! = b;^ 'X ♦Ln<9o>> ' 


(3.6.4) 


(3.6.5) 


(3.6.6) 

(3.6.7) 

(3.6.8) 


(3.6.9) 

(3.6.10) 

( 3 . 6 . 11 ) 

(3.6.12) 

(3.6.13) 


In,! “ n+il^> ' G(Xi,...,Xi+lJ} 


(3.6.14) 


and 
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since ^ ^ stationary, <|)“®^ixing seqiience of 

random variables , by Lemma 3.2,1, it follows that for 

fixed n, if i {3 ^fi >1} and (n , i > 1} 

' “ n , 1 . ' 

are also stationary and —mixing seguences of random 
variables, where 

(3.6.15) 

for k >5.^ and 1 otherv;ise. Furthermore by assumptions {A4) , 

{A6) , (3.5.2) and (3.6.4) it follows that the assumptions of 

Theorem 3,3.2 are satisfied by {Y {Z .} ? and 

n ; 1 n i 

(n^j^i) with f 0 < 1 < 1- Hence by iheoreia 3.3.2, 

we get 

®n 

(p)lim I Y„ . = 0 (3.6.16) 

n-^ i=fl 

^n 

(p)lim I Z . = 0 , (3.6,17) 

n-^ i=l ' 

and 

®n 

(p) lim 5! ’^n i “ ® ■ (3.6.18) 

n-H» i=l ' 

By assumptions (A3) and (A4) we have 
n-*°° o ' 

lim E (0 ) = - iO,.) 

6 1 ,n o o 

and 

lim E [G(X^,,,.,X^ )] = M < “. 

Therefore, by (3.6.9) - (3.6.11) and (3.6.16) - (3-6.18) 
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we get 

(p) lim B =0 
n-*°° 

(p)lim = -i(e ) 

and . 

(p)lim = H . 

n-Ko ^ 


{3.6,19) 

(3.6.20) 


(3.6.21) 


NOW, let e >0. Choose t > 0 such that {6:6 -r < 9 < 9 +t} 
is contained in V(e^). Let = {x ; > t^} - 

S2 = {X ; Ib^|+ 1 ( 0 ^)! > t); S3 = {?. ; {b2[ > M+ 1 } and 
S — S U S2 U S3 where S ' is the complement of the set S . 


Since B^, B^ and B 2 converge in probability, there 

exists an integer nQ(T,£) > 0 such that for n > 

P 9 q(Si) < e/3, Pg (S^) < e/3 and Pg (S 3 ) < e/3 so that 

P (S') < e and P (S) > 1-e. For 9 = 6 ±t, we have, by 
®o O 

{ 3 . 6 • 5 ) that 

8 / ' Cl? P„( 6 ) 

= Bq± B 3 ^t+B 2 T^. (3.6.22) 

For n > n^(T,e) , with probability exceeding 1-e, 

|Bo+T^rj^B 2 l < (M+2) and -B^t > (i(0^)-T)T. So if 

we choose t <i(9^) (M+3)~^ the sign of B^i 

depends on that of with probcibility greater than 1 -e for 

n > n^(T,e) . Therefore, 

1 ^ log pjj(9) > 0 for 0 = - T and 

log pj^(e) <0 for e = Oq + T 
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with probability greater than 1-e for n > n„(T,e). since 

~ log Pjj(6) is contimious, there exists a valno 0 of 0 

n 

such that 

log ® 

30 ri n 

A. 

for in I0 q-t, Q^+t] v^ith probability graater than 1-e 
and for n > n^(T,€) . Hence under assumptions of Section 3.5 
the approximate likelihood equation has a consistent root, 

Suppose the following two conditions are satisfied 
in addition to the assumptions of the previous section. 

(A7) For all 0 e 0, 

lim E log p (0)1^'^'^ = K{8) < <» 

for 0 < 6 < 1, 6 independent of 9. 

(A8) The mixing coefficient satisfies the condition 

(J,^ = O(n"0) {3.6.23) 

/■ 

for 0 > 1+2/6 where 5 is as in (A7) . 


Theorem 3.6.2 ; Under the assumptions of Section 3.5, (A7) 
and (A8) the approximate maximum likelihood estimator is 
asymptotically normal. 


Proof: If 0 is an MLE, then by the previous theorem 

— n 

it is consistent and from (3.6.5) we have 


S— log ° “ ° 

... < 3 . 6 . 24 ) 
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where and are as defined in {3.6.6) - {3.6.8). 

Ilierefore , 


{B„ (0„-6„) = 5^ , 


where 


? = { i(e )}“^/2 3 
n n o o‘ 


ana 


-1 


nn=-a(6o)} t3i+(0„-e„)r B,1 . 


(3.6.25) 


(3.6.26) 


(3.6.27) 


By (3.6.20)and (3,6,21), (p) lin = -i{G^) and 


n-H» 


(p) lim B 2 = M. Since 0^ is consistent, (p) lim (0^-8 ) = 0 


n-x® 
Hence , 


Tl-’HO 


(p)lira = 1. 

n->oo 


Now, consider 5^. By (3.6.6), 


^n = ^^n 


- 1/2 


•"n 

I (9 ) 

i,n o 


o “o 


(3.6.28) 




-’n 


xvhere 


^#3. n l^n 
3n 


s, = 3 ”^^^ 

1 ,n n 


y Tp! (0 ) 

^i,n o' 

1=1 


We have 

,2 


o 1 


(3.6.29) 


(3.6.30) 


(3.6.31) 
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6 


.1 (S (9 )^. (Q j, 

“ 1=1 ^ l/i' o 

2 

= ^ ^ (9 )^' (0 )1 + n 

^ ° i=l l,n ^0 q)1 + P^ 


where 


"1 ^ 

^ i=l l,n o 


Since = 0 for all n and i and since 

^’^i,n^®o^' ^ t stationa3T7 , -mixing sequence 

of random variables for fixed n , we get ;by Lemma 3,2.3, 
that 

1 •> ®n 

lp„l <2B; E[»'^(e)]2 J H4.I”>}V2. 

By,.Leiaina 3.2.1 , for all i. iherafore 


IPnl ; -_^(e^)]2 6^1 j” 1 ^1/:^ . 

Talcing limits on both sides as n ^ , by (3.S.23} and 
assumption (A4) , we get that 

lim p = 0 


4 A/2 


n-x» 
so that 


n 


q(G^) = lim E(S^ ) = lim E[<|'' (0 )]^ 

1 ^ n n*^ In ® 


n-H» 


3 


n 


Therefore 

q(6o) = i(0 } + g{0 ) 


+ 2 lim I [Ei>; „(9 )4',’ , „C8 )} 
n-H» i=l 1/^ o 1+1 ,n o 


(3.6.32) 
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8 


n 


where g(e^) = 2 1^ J El*' (9 (e JI. since 

n->«> 1=1 o x+i^ia o 

{x^ , n >1} IS stationary and ^-mixing sequence of random 
variables satisfying assumption (AS), by l^mma 3.2.1, we 
get that i > 1} for fixed n is stationary -mixing, 

where for k > £ and 1 otherwise. Hence, the 

doiJble sequence of random variables {y . } satisfi«="s 
assTjmption (i) of Theorem 3,4.2. E( ^ i^^ ~ ^ 


(3.6.30) which is assumption (iii) of Theorem 3.4.2. 


Assvimption (iv) follows from regularity condition (A7) . 

Thus all the conditions of Theorem 3,4.2 are satisfied by 

{Y„ 4 } . Hence it follows that 
n jrX 

(JOlim I . = N{0,1) (3.6.33) 

n-H» izrl 

2 

Since, lim E(S^ ) = q(0 ), we get 
jj-Hxj J- , n 


(X.)lim = N(0, q(0^)/i(0^)) . 
n-+oo 


(3.6.34) 


As E converges in distribution to N (0,q(G^) /i(0 )) and rj 
n o o n 

converges in probability to one, we get that converges 

in distribution to nomal with mean zero and variance 

q(0Q)/i(0^) by Lemma 3.2.4. Therefore, from (3.6.25) and 

(3.6.32), we get that -0^) converges in distribution 

n n ^ 1 

to normal with mean zero and variance {l+g{0Q)/i{0Q) }{i{0Q) } 
Hence the theorem follows. 


Definition 3.6.1 ; An estimator T^ of 0 is said to be 
asymptotically efficient of first-order, if there exist 
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two constants a and b{^0), not depending on the sample 
such that 


(p)lim| 

n->oo 


{I„(6)} 


- 1/2 


log 


P^ce) 


- b{l (6) } 

n 


- 1/2 


CT^-e) 


~ (3.6.35) 

V7here 

In'«> = - l^log p^(e)l. 

Under assumptions (A1 )-{.a 6)- a consistent A1!LS is 
asymptotically efficient of first order. 


Proof; We have 


-I^(e) = - Egl^^log p^(0)] 

^n 

I SJ^log.p. (e)] 

i=l 0 98^ 

by (3.5.4) and (3.5.5). Therefore ^ by stationarity 


Eg [^log P3^_^<6)1 

Therefore, by condition (A4) , 

-1 


lim - 6, I (0) = i{G) > 0. 

n-Ko 


(3.6.36) 

In view of (3.6.1), (3.6.7) , (3.6,36) and by the 


fact that 0 is a consistent estimator, the conclusion of 
n 

the theorem follows as in Theorem 2.4.3. 

3.7 Example 

In this section , we give an example which satisfies 
the assumptions of Section 3.5. 
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Example 3.7 . _ 1 ; Let {X^ , n >1} be a sequence of random 
variables such that (i) E(X^) = y (ii) var(X^) = 1 and 
(iii) Cov(Xj^,X^) = pi I where jp| < i. suppose that ^ for 
all n, the n— dimensional joint density function which 
depends on a single xinknown parameter y is given by 

{axpt-{ -e )r;^(x‘"’ -6<"> ) V 2 I 

...{3.7.1) 

where T denotes the transpose of the vector, x^“^ and 

are Ixn vectors and {E (X^) , . . . ,E (X^) } 

respectively/ and r“^ is the inverse of the variance - 
covariance matrix of X^^^ . ae elements of are given 
by (ii) and (iii). Then 



From (3.7,1) and from the variance -covariance matrix it 
follows that the sequence ^X^, n > 1} is stationary and 
-mixing with 


The problem is to obtain the asymptotic properties of 
A MT.R of u. We show that the conditions of Section 3.5 
are satisfied and thus establish weak consistency. 
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asymptotic normality and first-ordp-r 0 4 = 4 =- • 

^ jr U Iirst: order efficiency of amle 

U of p. By Theorem 2.5,1 of ihider-^nn roi 
n -«■ ur 4Uiaerson [2], V7e get that the 

conditional density function of given 
to be 


P,!p) = (2.) -1/2 a-1 e.p{-(2a2,-l(,^.,^,2, 


where 

c 


: = E(X^) + r, - r"^ 

n « l,n-l n-1 ~ S f 


= Var{X ) - r, r * T 

n ri' M „_-i i 


n 


1 1 tT 

1 ;i-l n-1 pij. 


C3.7.5) 


{3.7.6) 


^l,n-l lx (n-1) vector P^"^,...,p}. We 

have 


r, _ , rr^, 


1, n-1 ^n~l 


- -f “2 

= ip 4 p 


) 


p} 

i 

) 


{ 0, 0 p} (xta-l) - 


- £ 


= P + E (Vl),)- 

Therefore 

= y(l-p) + p x^^^^ (3.7.7) 

and 

= 1 - (3.7.3) 

Hence (3.7.4) becomes 

Pj^(u) = {2 tt)"^/^ (1-P^)"^ expt"{2(l-p^)} "^{x^-p (1-p) 

...(3.7.9) 
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Clearly , 

l^iog P^d-) = (l+p)-^ 

and 

2 

log p^{]i) = -- {l-p)/(l+p). 


(3.7.10) 

(3.7.11) 


Conditions (A2) , (A3) and {A4)- follow from (3.7.10) and 

(3.7.11) , Since log pj^(y) is independent of 

^1 ''^2 ^n (^5) vrill be satisfied with 

G (x^^ ,^2 / • • • r x^) = 0. For all choices of the seqt^nces 
of positive integers {£^} , condition (A6) follows from 

(3.7.11) . Ass\xraptions(A7) and (AS) follotv since 
lim E l|-log P_(y)|^ = 2/2/-rr. 

n-Ko ^ 

Since the convergence of p^ to zero is always faster than 
that of n“® for all 0 > 0 assiamption (A8) will be satisfied. 
Thus all the conditions of Section 3.5 and assumptions 
(A7) and (A8) are satisfied by this example. Hence, by 
Theorems 3,6,1, 3.6.2 and 3.6,3, it follows that the AMI£ 
of y is consistent, asymptotically normal and first-order 
efficient. 


3.8 Conclusions . 

In this chapter, we proved weak consistency, 
asymptotic normality and first-order efficiency of an 
AMLE for stationary ij)-raixing processes with some 
assumptions on the mixing coefficient (|i. It is easy to 
see from Definition 3.2.1 that , if = 0 for all n, then 
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(f-mixing condition reduces to independence. Hie 
assumptions of Section 3.5 win be satisfied if «e 
choose the sequence of non-negative integers {l } 
to be identically zero and mle in this case rLuces 
to the usual MLE. If = o for all n > m, then the 


sequence of random variables {X^, n > 1 } are 
m-dependent. Thus m-dependence is a particular case 
of 4>-inixing process. Hence if a sequence of stationary 
m-dependent random variables satisfies the conditions 
of Section 3.5, then the conclusions of Ibeorem 3.6.1 
to 3.6.3 hold in this case also. The example given in 


Section 2.5 satisfies all the assumptions of Section 3.5 

except assumption (A6) . For, in this case there exists 

no sequence of non-negative integers such that 

lim a = c finite and 
n-^ 


lim E.[{L. 

n->co ^ ao 


log i X, . 

n 


)}^] = 


i(9) 


n 



CHAPTER IV 


BIAXIMTO'I likelihood ESTIMATICaJ -pOT? 

DEPENDEHT RANDOM VARIABLES 

4 , 1 Introduction ; 

In the previous two ciiapters f we studied some 
large sample properties of maximum likelihood estimators 
when the observations are from a sequence of stationary 
random variables which are asymptotically independent in 
some sense. In this chapter, we obtain strong consistency, 
asymptotic normality and first-order efficiency of an MI£ 
for arbitrary stochastic processes satisfying some 
regularity conditions. The results in Loeve {28] regarding 
the strong law of large numbers and central limit theorem 
for dependent random variables are used to obtain strong 
consistency and asymptotic normality of MLE respectively. 
Bhat and Kulkarni [5] applied similar result of Loeve [28] 
for double sequences in a different problem. 

We state some preliminary results in Section 4.2. 

The regularity conditions to be satisfied by the 
n-dimensional joint density function are given in 
Section 4.3. As in Bar-Shalom {4], these conditions are 
expressed in terns of the probability density of 
observations conditioned upon all past observations . Ihe 
proofs of strong consistency ^ asymptotic normality and 
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first-order efficiency of MLE are given in Section 4.4. 

AS an application of these results , we obtain the 
Bernstein-vonMises theorem for dependent random variables, 
in Section 4.5. Section 4 . 6 contains some examples which 
satisfy the regularity conditions presented in Section 4.3. 

4.2 Preliminary Results 

In this section , V7e state some leitmas which will 
be used later in the chapter. 


Lemma 4.2.1; If the random variables X, , k = 1,2,..., 
- _2 ^ 

are such that T b Var X < <» with lim b = <», then 

“ n n 

n=l n-x» 


n 


n 


(a.s)lira b~^ I {Xj^-E{Xj^|X^ .,Xj^_ 3 ^)} = 0. (4.2.1) 

k=l 


n->oo 


Proof : For proof, we refer to Loeve 128] , p. 387. 


Lemma 4.2.2 ; Let X^ 2 ' •••be random variables 

centered at their expectations. Under the Liapunov’s 
condition 


^ I 

lim I 

n-HX! k=i 


2+5 = 0 , (4.2.2) 


if 


, 

n-»-co k=l 


lim 'I e|E' = 0 


(4i-2.3) 


and 


lira E|E- -E X2^|= 0 

n-K» ]<=i 


(4.2.4) 
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then 


a..)lim I X ) 

n->oo n nK 


N(0, 1) 


(4.2.5) 


where 


I " 

k=l 


and N(0, 1) is rhe normal distribution with mean zero 

and variance one and E • denotes the conditional expectation 

with respect to X + . . . + x 

no n > -1 * 

^ n 


Proofs For proof, we refer to Loeve [28] p. 377 . 


^ ‘ ^ ^ sequence of random 

variables satisfying the following conditions. 

(i) lira I Elx, |2+'5 = 0 , 

n-xo k=l ^ 

for some 5 > 0 , 

(ii) E(Xj^ 1 X 3 ^,.. = 0 

almost surely f and 

(iii) E(x 2 |x 3 ^,...,Xj^_^) = 

almost surely, where is independent of X^ ,. . . ,Xj^_ 2 ^ . 


Then, 


(£)lira n~^/^ s”^ J X, = N(0, 1) 
n-K. ° kil I' 


where 


s 2 = n~l 


I E Xj^ = n-1 ^ . 

:=1 ^ k=l 


l4 


(4.2.6) 


and N( 0 , 1 ) is the standard normal distribution. 
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= Since the Q-fiela generated by the random variable 
t ... t is contained in the s-(iela generated by 

the random variables X„ . . . , assnmptions (4.2.2) 


to (4.2.5) of Lemma 4.2.2 will be satisfied if „e take 

^nk ” ^ 1 < k £ n. Thns, by an 

application of Lemma 4.2.2, the required result follows 


I'3msia .4_.2_._4= If g is a continuous function mapping 

s-dimensional Euclidean space into itself with the 

property that, for every 0 such that I|0|I= l, 

0Vg(e) < O, then there exists a point O such that 

Util < 1 and g^(G) = 0, where 6' q{Q) = I B g {Q) 

t=l ^ ^ ‘ 

Proof: For proof, we refer to Aitchison and Silvey [1] , 
Lemma 2 . 

^mma 4.2.5 : Suppose {u^, n > 1} is a sequence of randcmi 
variables such that Uj^ ->■ u in distribution. Further 
suppose that {V^^, n > 1} is another sequence of randcm 
variables such that 

l°n - \l 1 l^'nl 

where tends to zero in probability as n teiuls to 
infinity. Then converges in distribution to U. 


Proof ; For proof, see Billingsley [8], p. 62. 

Lemma 4.2.6 ; Let ...,Xj^ be k random variables . Bien 


for all r > 1, 
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Elf X.|>^ < f E! 

* «... 


X, 


i=l 


t=l 


Proof: The leimna follows by repeated application of 
Cj. "inequality of Loeve [28] . 


4,3 Regularity Conditions : 

Let the set of dependent observations be 


y” = yn> (4.3.1) 

where y^^ are real valued random variables, with a known 

joint probability density function with respect to a 
0 -finite product measure This density function 

depends on a £ "dimensional unknown parameter B, that is 




(4.3.2) 


The true value 0° of the parameter G is estimated by a 
Bo rel -measurable function 0j^(y^) obtained by maxkmizing the 
likelihood function 
n 

pCy^le) = n p-(e) (4.3.3) 

k=i ^ 


where 

= p(yj,|y^”^ , 9) » 


(4.3.4) 


We denote log p, (G) by g(k,'0) , the first partial derivative 
of g(ky 6 ) with respect to by g^(k; 0 ) and the partial 
derivative of gy(h ; 0 ) with respect to by 
Suppose the following regularity conditions are satisfied 


for all k. 
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(El) The parameter space 6 is an open 
ji -dimensional Euclidean space E^. 


subset of the ' 


(E2) For almost all 
g^j^(k; 0 ) exists for u 
throughout © „ 


y and for all 8 e 0 ^ g^{k; 0 ) 
^ ~ and are continuous 


and 


(E3) For almost all yk-1 ^nd for all o e 0 

% 

Eg fg„(k,2) ly'^’h = 0, n=i I 

and 


(4.3.5) 


- ®o'9uv(>''2>ly’"^I =Eg[g^(k,-0) g^(It,e)|yk-l] 

“ (4.3.6) 

for u,v = 1 , . . , , £, where ( 0 ) is independent of 

Yir . . . • 


(E4) For all G e 0 
, n 

lira n”-^ I ( 0 ) 

n->a> i=l ~ 

and the £x£ matrix 


“uy'?)' 


< GO 


Z(0) 

is non -singular. 


((au^(O) )) 


(4.3.7) 


(E5) For all tG e 0, there exists a neighbourhood V( 8 ) 
such that for all O’ e v{0) 

for all u,v = 1, ^ where G(y^,0) > 0, 


(4.3.8) 
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kil = M(e) < „ 


M(G) is a constant and Lk denotA <5 

i • I £ aenotes the usual Euclidean 


norm 

in E . 



(E6) 

For all 

. 0e6 and for all u,v = 1 n 


(i) 

00 

I k”2 

k=l 

Var^[g^(k?e)] < co , 

(4.3.9) 

(ii) 

11 8 

H 

1 

Var^ (k ? 0 ) ] < 

(4.3.10) 

and 




(iii) 

1 k-2 

k=l 

Varg [G(y^ ;0) ] < oo. 

(4.3.11) 

The following 

condition is weaker than (E6) . 


(E6') 

For all 

0 e e, and for all u ,v = 1 ,. . . ^ £ 


(i) 

lim k~^ 
k“^ 

Varp[gu(k;6)] < c», 

(4,3,12) 

(ii) 

lim k~^ 
k-^“ 

Varg [gu^(k?0)] < 

(4.3.13) 

and 




(iii) 

lim k"^ 

k-j-oo 

Varg[G(y^;0)] < 

(4.3.14) 

(E7) 

For all 

0 e 0 and for all u=l,.,.,Jl 



lim E 1 g (k;0)l^'^'^ =0 {4.3.15) 

n-H» k=l 0 ^ ~ 

for some 6 > 0 , independent of 9 . 

The maximum likelihood estimate of 0 is defined as 
a solution of the system of likelihood equations 
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n 


30 


log p{y I 0) = 0, u » 1, 0 


u 


(4.3.16) 


in a neighbourhood of the true parameter value e» . 

‘>•'5 The Asymptotic Propertiog of ..r. . 

georem 4 ^ 4.1 = If assumptions (El) . (e 6) of Section 4.3 
are satisfied , than the system of likelihood equations 
(4.3.16) has a root, which is strongly consistent. 


Proof: For all t = 1 i, expanding log p(yh| e, 

about the true value o" of the parameter o‘ it follo>re 
that 




3_ log p(y^i0) = ^ log p{y^|o°) + 


I 


+ I (Qg-Q®) ■ 

s=l ® s 30^303 


^ logp(y“)0') 


... (4.4.1) 


where 0^ = o“ + , la^hl, s=l, 


The right side of (4.4,1) can be written as 
n n Jl 



It follows from (E5) and (E6) there exists a neighbourhood 
V(6- ) such that for 0' e V{6°), 

lgts(kp0') - gts(k ,0'’)! <l0'- f Ij^GCy^r-e”) 

where G(y^;0°) > 0 ahd 
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n 

(a.s)lim n”-*- I e [G(y^j6“) = 

n-»^ k=l 0 ^ 1 j: j 


(say} 


Clearly, 
n Z 




n Z 




< £ I 0-0° I " I G(y^,Q®) 

~ Z k=l 

< Z^ l0-8° 1^^ J_ G(y^?0'>) . 


k=l 


Hence we get 


^ log p(y^|6)= n"^ ? g^.(k,0O) 

k=l ^ 


(4,4.2) 


gts<k,9°)} 

s-1 k=l 


+ ^ |6_0‘’|^n~^ \ G(y^;60) 

~ X- krri 

(4.4.3) 

where [g^j <1 for all n. By (E 3 ) and (E 4 ) , it is clear 
that for any t = 1, . . . , £ , 

(a.s)liin n“l y E {g^. Jk ,e?:) I y^ •^} =0, (4,4.4) 

k=l e° ^ - “ 

n 


(a. s) lira n“^ ^ E_ o ^ 


n-H» k— 1 


(4.4.5) 


for t , s=l , and 
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(a.s) lim n"-L I E.o{G{Y^;e“) „ 

n-K» k=i S ~ 1 -^ j 

Therefore, by (E6) and Leinma 4.2.1, for t,s = l 

n 

(a.s)lim n"l V g^.(k, 0 '>) = o , 
n-^ k=l ■ ^ ~ 

(a.s)lim n ^ J g_{k,0°) = ^ 

n-»^ k=l ~ ts 

and 

1 ^ V 

(a.s)lim n"-^ I G(y ,“6°) =M 

n-wo k=l ~ ° ^ 


(4.4.6) 


(4.4.7) 


(4.4.8) 


(4.4.9) 


where = 0 ^ 3 ( 6 "). Hie Ixa matrix = i(e») is non 
singular by condition (E4) and hence positive definite. 
It follows that there is a x > 0 such that 


i I 

^ ^ '^t ~ vector u in such that 

t:=l s==l '-s s> 

ful^ = 1. Suppose e> 0 is given. Choose 6= 6(e) >0 
such that 


'-5 < e,{Q ; |0“9°lji < 5}C v{ 0 '') and 5 <--x{3£^(M+1)}'“^. 

... (4.4.10) 

Let and S^, t,s = 1, . . . ,£ be sets such that 

OD n 

= U {y : [n“^ I g. (k,8'>)! > 5^}, 

n«n k=l 

o 

sj ’ = U {y : |n-l J ^ ' 

n»n k=l . 

o 

cp Xi 

S, = U {y 5 n"^ I G(y^;9°) > M +1} . 

n«n^ ~ k=l 

o 


and 



By (4.4.7, an. (4.4.9, and .y the definition of alnost sure 

convergence, there exist positive numbers n^{e,t), n (e t s} 

and n3(£) such that, for all t,s = l n pfc(t), ^ ’ 

> if.. ..,36, P(S' ^)<-e/(3£j 

for Uq > n^(e,t), P(sj^^) < e/i 3 S?) for 
P(S3)< e /3 for nQ> n3(e). Then, for 


^o ^ and 


By 


> N(e, = n,aK{ n^(e,t); n^Ce.t.s,; n3(t,,t,s=l t, 

all the above ()l+f2+l, inequalities will be satisfied. 
(4.4 .3) f we have 

, 1 r ^ 

k^l 

= |n-lJ^gt(k,G«,.tJ^( 5 ^e«, {n -1 
+ (S^ I e- 0 ” I 2 n-l G(y’', 0 ",| 


- Jl "t"''?"' I* X' V«:i|n-^ X ^ts ^ “tsl 


GCy’^je",. (4.4.11) 

Let S = union of all the (£+£^+ 1 ) sets t=l 

f t,s = 1,...,,£ and S., . Then on S, if [ S-e” L < 5, 
for all t = 1 X, the right side of (4.4.11) is 

i * 

o 

< <s -f 'S£|o-.e°|^+ £2 |0-.e“l^ (M^+i) 

< 3 £^ (M^+ 1 ) (S^. 


{4.4.12) 
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Hence, by (4.4,10), if iG-eOj = 5 ^ then 

x» 



z z 


Jl gii ^ts + 3^^ {M^+ 1 ) 6 ^ 


^ z z 

!9“QM2 I I a; 
^ t=l s=l 


{0t-6^) (Q^-e^) 3 ^ 

7 " ^ r-^' - + srcM +1)6^ 

1 0-8 7^ !8-e“i^ ® 


6^t + 3£^(M^+1) 6^ < 0 


(4.4.13) 


Since 5 < t {3£^(M^+1)} It follows from Leima 4.2.4 
that there is a value 0 of 0 such that 

<~n. 's, 

37“ I ^t^^' ^n^ = ° (4.4.14) 

t k — 1 

for t = 1,... ,Z, and such that < 6 < e. Since, 

^ a 

SCO {y ; I (9 - 0 “)[n~l I g (k, 0 “)] < 0 } 

n=nQ " t=l ^ ^ k=l ^ 


for n > N(e) and for 6 as in (4,4.10), it follows that, 
for n > N(e) and for 6 as in (4.4,10), 

^ n 

Pt U {y : I (0.-0?) [n'^ Ig.(k?G°)] >0}] = P[Sl 
n=n^ ~ t=l ^ k=l ^ ~ " 

Z , , Z Z r.^. 

i I P(S<*^)) + I I P(s7) + PCSj) 

t=l t=l S=1 


< Ze/{3Z) + Z^e/{3Z) + e/3 = £ • 

Hence, by (4.4.13) and Lemma 4.2.4 we get that ^ if 6 is 

/v 

as in ( 4 . 4 . 10 ), then there is a value 6 ^^ of 9 such that 
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—log p(y |e^) =0 

almost surely for t and such that |8 -80{ < ,>? < c 

for large n. This proves the existence of a strongly 
consistent solution of the system of likelihood 
equations (4.3.16). 

Remark 4 . 4 . 1 g If the assumptions (El) - (e 5) and (E6*) 
instead of (E6) are satisfied, then by an argument 
similar to that given by Billingsley [8] in Markov case, 
it can be shown that , the MLE is weakly consistent. 

Theorem 4.4.2 ; Under the assumptions (El) - (E5) , (E6*) 
and (E7) , any weakly consistent root of the system of 
likelihood equations (4,3.16), is asymptotically normal. 

Proof: Let y(n) = {y^ (n) , y 2 (n) , y^(n)} be the random 

vector with components 

yt(n) = I g^(k,9°). - (4.4.15> 

k=l 

/V 

If is a weakly consistent solution of the likelihood 
equations, let u(n) = {u^(n) . ,Ujj^ (n) } be the random 

vector with components 

Ut(n) = n^/2 (0^j^-Ot) •• (4.4.16) 

To prove that the random vector converges in 

distribution to normal with mean vector zero and variance 
covariance matrix ^y standard Cramer-Wold 
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result [15] , it is enough to s 
,q^ of real numbers 


how that for any set 


^t=l ^ ^iO, 3^) 


where 


0 = 1 l O. a 

^ - L t s -t ■••Is ' 


t=l S=1 


(4.4.17) 


(4.4.18) 


We have 


"n = 4 'It yt'") = I f 


]c=l t=l 


= n-V2 


L ^l< 


where 


“k = I q. ■ 


(4.4.19) 


(4.4.20) 


Let, 9y. be the a-field generated by yi,...,y^, for k > 1. 
Then by (E3) and (E7) 




H I 


t=l S=1 


(4.4.21) 

(4.4.22) 


where is independent of 


I and 


lim I ,1 |2t5, 

n-vo5 2 K 


lim J E ,1 f q gt(k;6')|^'^® 

n->^«> k==l S t=l ^ ^ - 

lim n-(l+«/2) f I 

n-K)c> V=1 -f--! *" ? 


k=i t=i ^ a ^ ~ 
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Ihe last step follows from Lemma 4 2 k n,, 

^ i./.b. By assumption 

it follows thatp 


CE7) 


-(1+6/2) ^ 

>=o 

n-^«> k=l y ^ 


(4.4.23) 


It is clear from (4.4,21) - (4.4.2^) 

tnau the conditions of 

Lemma 4.2.3 are satisfied and hence 


(jC) lim [n s“^ 


n 


n-x» 


I 1 

^ k=l ^ 


N(0,1) 


(4.4.24) 


where 

2 


n n 


n 

I 

k=l 


E(u2) 


n 


T jLi £-32/ 

r.I I I I 


n 


k=l t=l s=l 


'3*. CTo ^^4^ 1 
'‘t -s ts^ 


»-4.25) 

2 2 

Since s^ tends to 3 in (4,4.18) as n -*- «= by condition (E4 ) 
we get from (4.4.24) that 

n 

(l)lim I u, =N(0,3^). (4.4.26) 

n-^ k=l ^ 

Therefore by Crams^r-Wold theorem [15] ^ it follows that the 
random vector y (n) converges in distribution to normal with 
mean vector zero and variance-covariance matrix that is, 

O' 


(£.)lim y(n) = N(0, Z ). 

*1 -V o 


(4.4.27) 


Let the neighbourhood V(0®) and the fiinction G be 
as in the preceeding theorem. If ^ is a consistent 

A . 

solution of the likelihood equation , thena^ lies 
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{0 :|e-0Mj, < 6} almost surely. Hence by (4.4.3), ( 4 . 4 . 15 ) 
and ( 4 . 4 . K ) we have that 

n 

y^fn) + I u^(u) [n -1 J g.,(k, 0 ")l 
s=l k=l ^ ' 


'~n™! I £* . 1 G(y^;e®)] =0 


k=l 


{4.4.28) 


for t 1,... , Jl, where l3^| < i. From (4.4.4) - (4.4.6) 
and the fact that [ 0^-"O'> | ^ ^ 0 a.s as n - it follows 
that 


y^(n) - Ug(n) + s^^in)) 


s==l 


= -3^r (4.4.29) 


where e^^in) , and y^ are sequences which tend to 
zero almost surely as n oo. Therefore, 

Iy(n) •- E u(n)l < e iu{n)l (4.4,30) 

X/ n £ 

where ->• 0 almost surely as n -»- «>. Hence, by Lenuna 4.2.5, 

the distributions of y(n) and E^ u(n) are asymptotically 

the same. Therefore, by (4.4.27) , it follows that 

{£) limC Su(n)}=N(0,2) 

O ~ ~ O 

SO that 

(£)lim {u(n)} = N(0,E“^) {4.4.31) 

n-x» 

Hence the theorem follows from (4,4.16). 
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As an application of the above theorem, we prove the 
following result; 


*nieorem 4.4.3 ; Under assumptions {El) ~ (E7) , 


(£)lim [2 {maxlijpCy^l e) 3°) }] = Xp 

n-^ 8e© ~ 8“ ^ s, 

•y 

where x7 is the central Chi-square distribution with £ 

Jo 

degrees of freedom. 


Proof ; By the mean value theorem, we have 

£ 

g(k; 9) = g(kj0°) + I (0^-6^) gt(k;e“) 

z z 


+ l3„8,3|e-e»(jG(y'^;0») (4.4.32) , 

\»/here G{y^?0°) is as in 'Theorem 4.4.1 .and jg 1< 1/6. Since 

n 

9 is a consistent root of the likelihood equation, (4.4.32) 
implies that 

t - 

2[g(k|a) - gCkyO®)] = 2 I (8.„8°) q,.(k?0°) 


+ I I (e -e^) ( 03 - 0 °) gts(k;9‘’) 

t:=l s~l ^ ^ ^ ^ 


+ 23 £^ l6~0° 1^ G(v^^i9°) . 
n ‘ ~ ~ ' s. 


Summing from k=l , . . . ,n on both sides we get 
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2 [log p(y^|0) - log p(y”}6“)] = 2 f {(0.-0°) J g (k;G°)} 

t=l ^ k=l t 

+ X I (es"9“) ! g. (k;e°)} 

t=l s=l k=l 


+ 23 le-'G®!^ I Gi’/ ;3°) 

^ ^ k=l 


(4.4.33) 


Since 


l0-e°lj I G(y^?0°) = ju(n)t^n"^/^ I G(y^;9°) 

^ ^ ^ k=l ~ ^ ^ k=l ~ 

by (4.4.16), it follows, by (4.4.9) and (4.4.31) that 


(t)lim [|e-0°i^ I G{y^;9°)] = 0 


(4.4.34) 


Nox*?, (4.4.33) can be written as 


-21og Aji = 2 u^(n) y^(n) 


£ 2- 1 II 

T Y t^\ ,, r„-l V 


+ I I u (n) u (n) [n’-^ I g (k?0°)] 
t=l S=1 ^ ^ 


+ 23 10-6®1^ I G(y^je°) 

~ ^ k=l 


(4.4,35) 


!?here log A = log p{y^ 0°) - log p{y“tG). ^Therefore, 


by (4.4,7) - (4.4.9) and by (4.4.34) it follows that 


(£)lim “21og A = (£)lim < u(n) , u(n) >. 

Tx-^ 


Hence the required result follows from (4.4.31). 


Now, in the following theorem, the conditions are 
obtained for strong consistency and asymptotic normality 
of an MT.H when the n-dimensional joint density ftinction 
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-pends on a single unknovm parameter 6. Wien £ = the 

regularity conditions (El) - {e 7) reduce to the following 
form. 

(Cl) The parameter space 0 is an open interval in the 
real line. 


(C2) For all k, 
k 


g(k,G) , i = 1,2 exist for almost all 
y^ and for all 6 s 0 and are continuous in 0 . 


(C3) For all k, for almost all and for all 0 e e 

(yk-ij ^ 0, 


and 




(4,4.36) 






where aj^(9) is independent of y^,... ,yj^„j_. 
(C4) lim n“l I ol(Q) = ±(8) 


Li w - ’ 

n-H» ]^=:3^ k 


(4.4.37) 

(4.4.38) 

(4.4.39) 


V7here i(0) is finite for all 6 e 0 . we shall further 
assume that i(G) is continuous and non-zero for all 0 e 0 


(CS) For all 6 e 0 and for all k, there exists. a 
neighbourhood V(0) , such that, for every 9’ e V(6) 

_ g 2 ■ 2 ' 

|-^g(k,0") -l_g(k^0)} < |9'_e| G(y^,e) (4.4.40) 

o V 3 0 


where G(y^,» G) > 0 , 
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and 

■J n Ir 1 

(a.s)liro I E '[G(y ,9Hy 1 = M{9} < =o. 
n-*^ k=l “ 

(C6) For all 0 e G 



Theorem 4.4.4 : if assumptions (Cl) - (C6) are satisfied, 
then the likelihood equation has a root which is strongly 
consistent. 


Proof ; Proof of this theorem follows from Theorem 4.4.1. 

Theorem 4,4.5 ; Under the assumptions (Cl) - (C7) , a 
consistent MLE is asymptotically normal. 

Proof ; Proof follows from Theorem 4.4.2. In fact, if 9 q 

/V. 

is the true parameter value of 6 and 6_ is an MLE, then 

Xi 



119 


it follov/s that 

(£)lim ) = M{ 0 , i“^) 

11-K50 ^ o 

where i^ = KO^) . 

In this case one can prove the following result: 

Iheoreni 4,4,6 ; Under the assumptions {Cl) - (C 6 ) , an MIJE 
is asymptotically efficient of first-order. 

Proof; In view of assumption (C4) , the theorem follows as 
in Theorem 2 . 4 „ 3 . 

4 .5 The Bemstein-vonMises Theorem 

Suppose that in addition to (Cl) - {C7) , the 
following conditions are also satisfied. Hereafter we 
write hj^(e) = g{k; 0 ). 

(C8) A is a prior probability measure on (0, ^) , where 3^ 
is the a-field of Bore 1 subsets of 0 , A has a density X 
with respect to the Lebesgue measure. The prior density X 
is continuous and positive in an open neighbourhood of 
the true parameter 0 ^. 

(C9) Let, for each 6 e 0 and any e > 0, 

P^(0,e) = [sup{h^(e') - h^{0)} ; l0'-0l > s, 0' e 0). 

Then 

, n 

(a.s)lim n"-^ \ E [% (0,£)ly 1 is a finite negative 

n-^ k=l 
quantity and 

I k “^ Var 0 [Rj ^(0 ,e)] < «=. 
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(CIO) Let 8 denote the true parameter and P = Pq . Let 
o o 8 q 

H be a non -negative meas^lrable function satisfying the 
following condition = Ihere exists a number e , 0 < e < i^ 
such that 

OD 

B(0 ) = {i^/(2TT))^/^ / H(t) exp{-(iQ-c)t2/2} dt 

^ ' .-CO 

is finite, and for every h > 0 and eveiy 6 >0 

{a,s)liin [axp(-5n) / H{tn^^^)X(9 +t}dtl = 0 

n^ lti>h 

where 9 denotes an i-lLE, 
n 

We denote by p^COjy^) = Pjj{ 0 ly 3 _ . /y^) / the 

posterior density based on the observations y, . . . ,y 

1 ' n 

corresponding to the prior probability density X. Also, 
1st 

P^Uly"") = (4.5.1) 

so that p*{tjy^) is the posterior density of n^'^^(0-8^). 
If f(8) is a function differentiable twice with respect 
to 0, VJQ denote the first and second derivatives of f(6) 
by f’(Q) and f'VCG) respectively^ in this section. 

/N, 

In Section 4.4, it was shown that a MLE 0 has the 
' n 

following properties , under the assumptions (Cl) - (C7) 

(i) (a.s)lim 0 = 0 , (4.5.2) 

n^co “ ^ 

(ii) i_ log p(y^l0„) = 0 a.s for n > N, 

30 n 


(4.5.3) 
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and ! 

(iii) {£)liin = MCO, • {4.5.4) 

We shall now prove the following theorem which may 
be regarded as a generalisation of the Bemstein-vonHises 
theorem for dependant random variables. The proofs of the 
results of this section are similar to those of Borwanker, 
Kallianpur and Prakasa Rao [11] , We define 

n ^ ^ ^ 

Vj^(t) = exp[ I {hj^(0^+tn~^/2) (4.5.5) 

lc~l 

and 

CO 

^n " / tn“^/^)dt, (4.5.6) 

Then from (4.5,1), it follows that 

PJCtly"^) = v^(t) X{0^+tn“^/^) . (4.5.7) 

Theorem 4 . 5 . 1 g Under assumptions (Cl) - (CIO) of 
Section 4.3, 

(a.s)lim [ / H(t) jp*(tjy^) -(iQ/27T)^'^^exp(-i^t2/2) jdt] =0. 

v«.co 

The proof of this result is based on the following lemmas . 
Lemma 4.5.1 ; Let assiamptions (Cl) ~ (C?) and (C8) be 
satisfied. Then the following conclusions hold: 

(i) for every e{0 < e < i^) , there exists a 6^ > 0 and 
an integer N such that 

{-(iQ~e)t^/2} 

for ! t 1 <6 n^/^ and n > N ; 


(4.5.8) 
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Cii) for every 6 > 0, there exist.' a positive s and 
an integer N such that 

sup, V (t) < exp(-ne/4) (4.5.9) 

it|>6n-^^ ^ 

for n > Nf and 

(iii) for every fixed t ^ 

2 

(a.s)lim Vj^{t) = exp (-it /2) . (4.5.10) 


Proof ; We have 


n 


log v^(t) = I -hj,(0^)} 

JL 


= tn 


- 1/2 


n 


n 


I h^(e ) + t^/(2n) I h" (e!) 

k=i ^ ^ k=i ^ ^ 

...(4.5.11) 

where |0^-9^j < tn The first term on the right hand 

side of (4.5,11) equals zero a.s for n > (say), by 
(4.5,3), For the second term we have 

t2/(2n) I h"(0M = t^/(2n) f 


n 


+ t2/(2n) I {h"(6^) -h"(8^)}. (4.5.12) 


k=l 


Since the first term on the right hand side of (4.5.12) 

O 

converges almost surely to -i^t /2 by (C4) , (C6) and 
Lemma 4.2.1 it follows that for a positive e, (e < i^) 


n 


t2/(2n) I h^(0^) < (-i^+e/2)tV2 


k-1 


(4.5.13) 


for n > N 2 , where we may take N 2 > N^. As 0^ is a 


MLE 
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which is strongly consistent by Theorem 4,4.1, v/e can 


choose a positive 6^ such that 0 < 6^ < e/CSM^) , 

f®n”®6i '^o f^n"^ni i ^ <6 for n > (say). 

o 

Hence, xf n > , 1 Qj^“9q| < 26^ and by (CS) , 



which tends almost surely to 26^14^ by (4.4.11). Combining 
(4.5.11) - (4.5.14) we get 


log v_^(t) < t 2/2 (4.5.15; 

for |t| < and n > tJ = ikix(Ni ). This proves 

(4.5.8). To prove (4,5.9), V 7 e have 

n i log v^(t) = n-1 "^k ^ V ^ 

(4.5.16) 

If iOn“®ol n , then jtn-V2j > g implies 

that j 0^+tn > 6/2. Hence for n > , 

n 

+tn-l/2j -h (e„)) 

]c=l o 


S I [sup{hk.(0)-h^(O.)} : |o~9^| > s/ 2 ] 

= Jl '*k<eo-«/2) (4.5.17) 

by (4,3.16) . By (C9) and Lemma 4.2,1, v/e cret that 
'1 ^ 

,1- converges almost surely to a finite 

i^ssjL 
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negative quantity. Hierefore, for n > 


n 


^ I {\(e^+tn"^/2j -hj^Oo)} < 0. 
^ — 1 


(4.5.18) 


Moreover 

1 

-1 




I h"(6') (4.5.19) 

which, by arguments given in (4.5.11) - (4.5.14) and by the 
fact that 8 ^ converges a.s to 0 ^ as n ->• «> converges* a«s to 
2 ero as n choose e such that 

n 

0 < e <-(a.s)lim n**l I E [% (6 6 / 2 ) j y^-l] . ( 4 . 5 . 20 ) 

n-x» k=l o 

Combining (4.5,16) and (4.5.18) - (4.5.20) it follows that, 

log V^(t) < -ne/4 (4.5.21) 

for [t] > n^/^5 and n > W. = max(Nj^, N 2 , N 3 , N^^) , which 
proves (4,5.9). Next, for a fixed t and e > 0, choose an 
such that 0 < t2/2 < e. By (4.5.11) ~ (4.5.14), we 

get that 


log v^(t) + t^ Iq/^I ^ ^ 1 /^ 


(4.5.22) 


for n > max(N ^ t^/5^) . (4,5.10) is thus proved. 


In view of the above lemma , the proofs of the 
following two lemmas are the same as those of Lemma 3.2 
and Lemma 3.3 of Borwanker , Kallianpur and Prakasa Rao [11]. 
But we give the proofs here for completeness. 
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Lemma 4.5.2 ; Under the assumptions (Cl) - (CIO) , there 
exists a positive 6^ such that 

(a.s)limE / H(t)jv (t) X (6 +tn“^/^) -A (6 ) exp(-i t2/2) Idt] 

\t\<6^n^/^ n n o 

= 0 . 

Proof : It is easy to see that 

/ H(t3 I V (t)X(6 +tn"^/2j_^^Q 2/2) jdt 

ltl<5Qnl/2 o - o < 

- J, 1/7 

ltj<6^n-*-^^ 

T/o H(t)v (t)lX(6o)-X(0 +tn-l/2)j<at. (4.5.23) 
t <6 nl/2 n n 

1 1 “ o 

Choose an e > 0 such that 

/ H(t) exp[''-(i “e)t^/2]dt 

—oo 

is finite. This is possible because of (CIO). Then there 
exist <5 and an N such that 

v^(t) < exp [-(i^-’e) t2/2] 

for |tl <5^ and n > N , by Lemma 4.5.1. Hence, by 

(4.5.22) , v?e have 

(aoS)lim[ / H(t) X(e ) I V (t) -exp(-i t^/2) 1 dt] = 0 

n-^ !tl<6^nl/2 on o 


by the dominated convergence theorem. Further, 
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/ H(t)v (t)|X(0 )-X{e +tn"^/2)j dt 

}t[;^6^nV2 n o n 

< sup { X(e)~X(8.) 1 E / H{t)axpE-(i -e)t2/23dt] . 

\e~Qo\^h {t|<6^nV2 

For a given ri, choose 6^ < 6]^ such that 

sup j X(e)-X(8^)| / T ,, H(t)exp[-(i -e>tV2]dt 

8-6^ <5 ^ lti<6 n-*-/^ ° 

< n . (4.5.24) 

Combining (4.5,23) and (4.5.24) ^ we get the required lemma. 

Lemma 4.5.3 ; Under assumptions (Cl) - (CIO), for every 6 > 0, 

(a.s)lim [ J H(t) Iv (t)X(8 +tn"^/2)-X(6 )exp(-i t^/2) jdt) 

n-x» |tj>6n-‘-'^ n n o o ' 

= 0, 

Proof % It is clear that the term in brackets on the left 
s ide is 

< / H(t) V (t) X(0_+tn"^/2) 

1 1 j >6n-‘-/^ ^ 

+ J , H(t) X(e ) exp(-i t2/2) dt 
|ti> Sni/2 o o 

< exp(”ne/4) / H(t) X(8 +tn"^/^) dt 

|tl>6nl/2 

+ / I/O exp(-i t^/2) dt (4.5.25) 

ltl>6n-‘-/ 

by (4.5.9). By assxjmption (CIO) , the right side of (4.5,25) 
tends to zero a.s as n Hence the T-erama follows. 
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Now, we prove Theorem 4.5,1. From Lemnas 4.5.2 and 
4.5.3, we obtain that 

CO 

{a.s)lim / H(t) j Vj^{t) X {0^+tn"^/^) -A (0Q)exp{-iQt^/2) j dt = 0. 

(4.5.26) 

Putting H{t) = 1, which satisfies the assumptions on the 
function H trivially , we get that 

oo oo 

Cn = /vn(t) X(9j^+tn"^/2) dt ^ XCG^) / exp(-i^t2/2) dt 

••-OO ■"■CO 

= X{6q) (2TT/i^)^/2. (4.5.27) 

Hence , by (4,5.6), 

OO 

/ H(t) lp*(tly^) -(i^/(2TT) )^/^ exp(-i^t^/2) [dt 

CX> 

< / H(t)|c^^ 1 (6„+tn“^'^^) Vj^(t) -C~^X{0 )exp(-i t^/2) jdt 

..»00 -1 U o 

00 

+ _/H{t)|c;;l X{ 8 Q)-(i^/( 27 r))^/ 2 |exp(-i^t 2 / 2 )dt. 

Hence Theorem 4,5,1 follows by (4,5,26) and (4,5.27), 

As a corollary to Theorem 4.5.1, wa give below a 
result which includes, besides the more traditional form of 
the theorera of Bernstein-vonMises, other interesting 
variations , 

Theorem 4.5,2 ; Let assumptions (Cl) - (C7) be satisfied. Let 
the prior density X satisfy assumption (C8) and 

00 

/ |ei^X(0) de 

—00 


< 00 


(4.5.28) 



for some non -negative integer m. Then 


(a.s)lim / [t j^ Ip* (tjy^) ~{iQ/(27r) exp ( -i^tVs ) | dt = 0. 

Xl-^oo —oo ^ 

(4,5.29) 

Proof s For m > 1, set H(0> = j9|"* in Theorem 4.5.1, From 
C^-inequality , we have for positive ntmbers h and 6 

Q-Sn j H{tn^/^) X{0-+t) dt 
Itl>h 

= n^/^ / X{t)|t-0 j’® dt 

lt-9^1>h ^ 

< 2^“^ n^/^ [ / X(t)lt!”^ dt + lij”® / X(t)dt] 

|t-9^i>h ^ lt\l>h 

_ CO 

< 2^“^ n^/^ [ / [tl"^ X(t)dt + le 1^] (4.5,30) 

-CO ^ 

which tends to zero a.s by Theorem 4.4,1 and assumption 
(4.5.28). It easily follows that 3(0^) in (CIO) is finite 
for H(t) = jt]’^. Hence (CIO) is satisfied. Thus the 
conclusion of Theorem 4.5.2 follows from Theorem 4.5.1. 

For m = 0 the assertion of Theorem 4.5.2 is the 
classical form of the Herns tein-vonMises theorem. 

Theorem 4.5,2 is an extension to arbitrary stochastic 
process of the corresponding result for the case of Markov 
processes due to Borwanker , Kallianpur and Prakasa Rao [II] . 

As an application of Theorem 4.5.1 to the theory of 
asymptotic Bayesian inference for stochastic process, we 
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shall derive results similar to those in Bon^anker, Kallianpur 
Kallianpur and Prakasa Rao [11] , 

Following LeCara [25] f we define a regular Bayes 
estimate estimate which minimizes 

Bn(B) = /£(e,a) p^{9ly^) de 

for all (yifo.»?y^) and all n^ where £{0,g) is a loss 
function defined on 0x0, We assume that a measurable, 
regular Bayes estimate of 6 exists. 

Theorem 4,5,3 ; Let , n > 1} be a process satisfying 
assumptions (Cl) - (CIO) , Let be a measurable regular 
Bayes estimate of 6 v/ith respect to a loss function £{9,3) 
which satisfies the following conditions. 

(i) £(9 ,8) = £(6-8) > 0, 

(ii) ^(t^) > £(t 2 ) if tj^ > t 2 > 0 or if t^ < t 2 < 

There exist constants {a^} and functions K(t) and G(t) 
such that 

(iii) a >0, 

n - 

(iv) G(t) satisfies assumption (CIO) and a^^ £(tn“^'^^) < G(t) 
for all n , 

(v) a^^ £(tn'"^/^) K(t) uniformly on compact sets, as n-+“. 

(vi) / K(t+m) exp(TiQt^/2) dt has a strict minimxam at m = 0. 
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'Rien 

(i) (a,s}lim (T “9 ) = 0, (4.5. 31) 

n-Ko ^ ° 

(ii) (DliiErtvie^-T^) = N(0, i"^) , (4.5«32) 

n-+“ 

and 

(iii) {a.s)lim a_ B„ (T ) = E{t) exp(”i_t^/2)dt . 

^ ii n n o o 

Proof g In vievr of Theorem 4.5.1, the proof of this theorem 
is the same as that of Theorem 4.1 of Borv/anker , Kallianpur 
and Prakasa Rao [11] . 

Remark 4 . 4 . 3 g Chao [13] showed that the maximum likelihood 
estimator 8^^ and the Bayes estimator 9^^ are asymptotically 
equivalent under some assumptions on the likelihood function 
and the loss function, when the observations are independent 
and identically distributed. Similar results can be obtained 
for arbitrary stochastic processes satisfying conditions 
of Section 4.3. 

4 .6 Examples g 

In this section, we present two examples which 
satisfy the conditions (Cl) - (C7) of Section 4.3 and thus 
establish strong consistency , asymptotic normality and 
first-order efficiency of MLEs of unknown parameters. The 
first example is a first-order auto-regressive scheme, 
which forms a non -stationary Markov process and the second 
example is a non -stationary , equi-correlated stochastic 
process . 
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Example 4 « 6 . 1 5 Let the sequence of random variables 
n > 1 } satisfy the relation 

^ (4.6.1) 

where (a[ <1 is known and the sequence of random variables 
^^n^ ^ independent , normally distributed with 

unknown mean y and variance unity. Further , let u^ = 0 
for n < 0, From (4.5.1) we gat 


^ “n-i+l 

r=l 

so that 

E = yCl-a"^) (l-a)~^ 
and for n > m 


(4.6.2) 


(4.6.3) 


Cov(X^,X^) = (1-a^®^) (l>ry2)"^ 


(4.6,4) 


Since the nxn variance -covariance matrix r of (X^,.,.,X^) 
is given by (4.5.4) , we have 


Y~1 

n 


l+a‘ 


■-•a 


~a l+a'' 


-a 


0 

-a 

1+a" 


0 

0 




•a 


(4.6.5) 


Eet denote the variance-covariance matrix of the 


ix(n-l) random vector X 


(n- 1 ) 


— (•^'i r • • • fX—^-i } and r. 

l,n“l 


be 



the Ix(n-l) vector let (i a ; ^ a v-*- > i 

By Theorem 2,5.1 of Anderson [2], the conditional density 


(n~l) = 


IS 


■n 
where 


function of given X 

p„{y) = (211) 0-^ exp[-(2d2)-l (x^^^)2] (4.6.6) 

,..{4.6.7) 

(n-l) = {E(X^), E(X 2 ),..., E(X^_ 3 ^)}, (4.6.8) 


2v -1 


= E(X^) + (l-ad ^ ^i^n-1 ^n-l^~ 


and 


2 X 2v"-2 ^ -1 t,T ^ (4.6.9) 

D‘ = Var(X^) - (1-a ) ^n-1 n-1 ^1^-1 


It is clear thatj 


2 ,, . 2n-2, .. 2, -1 


„ 2x-2 p p-l, r:: T = a (1-a ) (1-a ) 

(1-a ) n-1 ^n-1 d ^-1 

' ...(4.6.10) 


and 


•>1 r,-l , (n-DxT 

Cn = ^ 41-1 Vl ’ 


(4.6.11) 


wher 


, o -1 -1 , (n-lKT 

r = y(l-a)~ (l-a“) - (1-a ) ^1^-1 ^n-1 ~ 

n 

= li(l-o)"^ (l-a“) - C1E(X^_J^) 

= y(l-a)-l d-c") - wCl-^)-’- d-a''-^ = V -(4.6.12) 
Therefore, by (4.6.3) - (4.6.12), it follows that 


= t + “ ==1.-1 


and 


= 1 
n 


(4.6.13) 

(4.6.14) 
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Therefore, 

p^(y) = (27r)"l/2 [ -(x^~y-ax^_^) 2/2] 


(4.6.15) 


so that 

log p„(u) = - y -ax^_^ 

cind 


(4.6.16) 


^ log p^(u) = -1 . (4.6.17) 

Asstomptions (Cl) aiid (C2) follow trivially and to verify 
assxiraption (C3) , we observe that 


^ f 3y *’n 1 

Xi,, .. /X^_^3 


= E^IXjj-(u+Xn_i) 

[ 

1 Xi, , , . = 0 

(4.6.18) 

E {{ ^log p (y) 
y 9y “ 

1 Xj^ / . . . 



- [{Xj^" (u+Xj^ j^) } 1 • • • / 

= =1 (4.6.19) 


by (4.6.13) and (4.6.14). Thus (C3) and hence {C4) follows 

r.2 

from (4.6.19) with i(y) = 1 for all y. Since i— log p (y) 

5 y 2 n 

is independent of (C5) will be satisfied with 

G(x 3 ^, .. .,x^;y) =0. By (4.6.18) , (4.6.19) and by 
assumptions (C4) and (C5) , the validity of CC6) follows. 

The conditional distribution of ^ log given 

j^(n-l) _ ^(n~l) normal with mecin y+x^_j^ and variance 

on« by (4.6.15). Therefore 

E [|i_ log p (M) 1 ^ |x<n-U = = 2 / 2 /ir. 


( 4 . 6 . 20 ) 
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It follows from (4.6.20) that- {r'r\ i.- , 

.u.iij; rnat (C7) xs satisfied, -mus all 

the assumptions of Theorems 4.4.4, 4.4.5 and 4.4.6 are 

satisfied and hence the MLE of p is strongly consistent, 

asymptotically normal and first^rder efficient. 

Let {X^, n>l}bea sequence of random 
variables such that (i) e(X^) = p or 2y according as n 
is odd or evenp (ii) Cov(X^, X^) = c for all n and m (n?%) , 
and (iii) 7ar(X^) = 2C for all n. Further we suppose 
that C is known and y is unknown. The problem is to obtain 
the asymptotic properties of a MLE of y. Suppose that, 
for all n, the n-dimensional joint density function is 
given by pCx^^, . . . ,Xj^jy) 


= (2Tr) "^/^ {det exp[-{ (x^^^ -e -e '^}/2] 


... (4.6.21) 


where x^^^ and e are l^n vectors {x, ,.,.,x„} and 
{E (Xj^) , . . . , E (X^) } respectively , and r“^ is the inverse 
of the variance -covariance matrix of the random 
vector X^^^ . Here 


n -1 -1 . . . -1 

“**JL xi ““i o » © '“"i 


r"^= {C(n+1)}"‘ 

n 


( 4 . 6 . 22 ) 


L”1 ^ ^ 

As in Example 4.6.1 , by 'Rieorem 2.5.1 of Anderson [2], 
we get that the conditional density of X^^ given 
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^{n 1 ) _ y,in-l) 


P,(>i) = (2Tr)-°''2 D-1 exp[-{2:^rl <=Si-<^n>^l (4.6.23) 

n-1 


n 

where 


i=l 


and 


= Var(X^) - n-1 C{n-1) , 

Hence, by (i) , (ii) and (iii) , we get that 


(4,6.24) 


(4.6,25) 


n-1 


r(2n)~l (8-n)y + n”! I x. if n i 


^n = -5 


IS even 


i=l 


n-1 


(4.6.26) 


L(2n) ^ (5-3n)y + n ^ T x. if n is odd 

i=l 2. 


and 


c= c n"l (n+1) , 
n 


(4.6.27) 


Therefore, 


n-1 


log p_ (y) = - — log 27 r - - — 2 (x -r y-n“l ^ x . ) 

2 2C(n+l) “ " i=l ^ 

...{4.6.28) 

vvhere r =(2n) “I {8-n) or (2n)“l (5-3n) according as n is 
n 

even or odd. Since 

nr 


3U 

and 


log p^(y) = 


n 


(* -r p-n-^ I Xj_) 


n-1 


C(n+1) n n 


3^ 


nr‘ 


9y 


log p„ (li) 


n 


(4.6.29) 


(4.6.30) 


2 ^ -n C(n+1) 


Assumptions (Cl) - (C4) and (C6) are easy to verify and 
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follow as in Example 4.6.1. since -L. loq d (u) is 

8y2 ^n 

independent of , (CS) will be satisfied with 

G{x^j , o . fX^^y) =0, As in Example 4.6.1, {C7) follovjs. 
From (4.6.30) we get 


^ 3U ^ 


k 

2 _ k ^ 


:{k+l) 


Hence at are uniformly bounded by 4/C. Therefore, 

_1 n 2 . 

i(0) = lim n *^ 1 ^ is finite if it exists. In fact 

n-^ k=l ^ 

it can be shown in the following way that i(0) = 1/4C. 
We have 


3- 


Ijj(v) = E [ — - log p{Xj^, .. .,x ,-u) 
^ aii^ 


n 


= I E log p, (U) 


VI 


k=l 


3U' 


a2 , 

= I E [E ^ log pj^(y) |x 3 _,...,x^^ 3 ^] 


k=l ^ 


^ 2 
= y E [» o . 

k=l VI k 

0 

since cr^ is independent of we get 

JC 


n 


I (y) = 
n 


I- 4 • 


k=l 


Therefore, 


lim n“^ I a? = lim n“^ I^CP) = 


Jl^CO 


k=l 




We have log p(Xj^, . . • 


n n 


(4.6.31) 


n 


log 


2 x- 1 log(det F ) “ ,1, .1 (x 

9 n 2 1*1 j=i 


p) 


...(4.6 32) 
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where yij are the elements of r;l to 1 or 2 

according as n Is odd or even. Pro»> (.-,.6.32) we get 

f^logp(K^,...,x„,„) = - j j _ 

1—1 3=1 J 

After further simplification, it can be shown that 


In(y) 


"n{n+5) {4C(n+l) 

^ (n^+lOn-7) {4C(n+l) 


Hence it follows that 


if n is even 


if n is odd. 


i{y) = lim n"^ = 1/4C. 

n-K» 

Thus all the assumptions of Theorems 4.4.4, 4.4.5 and 4.4.6 
are satisfied and hence it follows that the maximum likelihood 
likelihood estimator of y is strongly consistent , 
asymptotically normal and first-order efficient. 

4.7 Conclusions i 

In this chapter, we obtained strong ccaisistency, 
asymptotic normality and first-order efficiency of HLE 
when the observations are from arbitrary stochastic 
process, m- dependence and <})-mixing treated in the previous 
two chapters are special cases of stochastic processes, 
but the set of conditions of Section 4.3 of this chapter, 
will not imply the conditions assumed for them. Hcw^ever, 
the examples presented in the previous two chapters will 
satisfy the assumptions of this chapter. 



CHAPTER V 


suggestions for further work 

In the previous chapters , the consistency either 
strong or weak, asymptotic normality and asymptotic 
efficiency of first-order are proved when the observations 
are from dependent random variables. When the observations 
are independent , all the three sets of conditions presented 
in Sections 2.3, 3.5 and 4.3 reduce to Cran^r’s [141 
conditions, of which some may be superfluous. Thus we note 
that the results obtained in these pages are natural 
extensions to dependent random variables, of the existing 
results for independent random variables . Except for this 
fact, the results of Chapter II and Chapter lil cannot be 
compared with any other results as there are none in this 
direction till now. Hox«7ever, the results about consistency 
in Chapter III may be compared with those of Bar-Shal<xn [4] 
for arbitrary stochastic processes. These results may be 
further improved as pointed out below. 

The regularity conditions presented in Sections. 2 ,3 , 
3 . 5 and 4 . 3 may be relaxed so as to cover a wider class of 
density functions . 

Conditions may be established which ensure strong 
consistency and asymptotic normality of MLE in the 
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m-aapendent case and strong consistency in <^ixing 

case. 


Ibragimov [20] has proved the following theorem; 
For a sequence {Xj , j > 1} of stationary ^-^lixing randan 
variables, if lim = o, ECX^) = o, 


some 0 < S < 1 and lim E( j x = ..then there exists 

n-xa i=X 


a constant C such that 

E i I < C{E{ ^ 

i=l i=l 1 

The extension of this result to double sequences of 
stationary ^-mixing random variables is very much useful - 
If this result is obtained, the assianption on the rate 
of convergence to zero of the mixing coefficient can be 
removed . Moreover , the proofs of the results of 
Chapiter III can be simplified. 


Since theorems on asymptotic behaviour of estimators 
are useful for practical purposes, only when the rates of 
convergence are known , some work in this area has been done 
by Pfanzagl [33] in the independent case and Prakasa Rao I3ff] 
for Markov processes. Similar results may be obtained for the 
general cases discussed in the previous chapter. 

Billingsley [8] and Sarma [451 studied some asymptotic 
properties of the test statistics where the random variables 
are Markov dependent . Similar results have to be obtained 
for the cases treated in the previous chapters. 
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